Multiple modules in a single pyd

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Multiple modules in a single pyd

Olivier Voyer
Hi everyone,

Is it possible to have multiple modules linking to one single pyd file? I'm using SWIG with VS2010 and I can't find a way of doing that.

Thank you,

Olivier

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Jim Bosch-2

On Nov 15, 2011 8:53 AM, "Olivier Voyer" <[hidden email]> wrote:
>
> Hi everyone,
>
> Is it possible to have multiple modules linking to one single pyd file? I'm using SWIG with VS2010 and I can't find a way of doing that.
>

I believe this is not supported by the Python C-API itself, regardless of what wrapper generator or library you use.  The name of the module import function needs to be related to the loadable module file name, and you generally can't have two functions for which that's true.

Jim


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Jérôme Laheurte

Le 15 nov. 2011 à 15:24, Jim Bosch a écrit :

> On Nov 15, 2011 8:53 AM, "Olivier Voyer" <[hidden email]> wrote:
> >
> > Hi everyone,
> >
> > Is it possible to have multiple modules linking to one single pyd file? I'm using SWIG with VS2010 and I can't find a way of doing that.
> >
>
> I believe this is not supported by the Python C-API itself, regardless of what wrapper generator or library you use.  The name of the module import function needs to be related to the loadable module file name, and you generally can't have two functions for which that's true.

But you can put several submodules in an extension module, as a workaround (see code below). I always wondered if that was possible with boost::python or SWIG ?


#include <Python.h>

static PyObject *myfunc(PyObject *self, PyObject *args)
{
    if (!PyArg_ParseTuple(args, "|myfunc"))
        return NULL;

    printf("Hello, world\n");

    Py_INCREF(Py_None);
    return Py_None;
}

static PyMethodDef functions[] = {
    { "myfunc", (PyCFunction)myfunc, METH_VARARGS, "" },
    { NULL }
};

static PyMethodDef nofunctions[] = {
    { NULL }
};

void inittestmod()
{
    PyObject *mdl, *submdl;

    if (!(mdl = Py_InitModule3("testmod", nofunctions, "")))
        return;

    if (!(submdl = Py_InitModule3("foo", functions, "")))
        return;

    Py_INCREF(submdl);
    PyModule_AddObject(mdl, "foo", submdl);
}

After building,

>>> import testmod
>>> testmod.foo.myfunc()
Hello, world
>>>

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Olivier Voyer
In reply to this post by Jim Bosch-2
Jim, thank you for your answer.

What if I have this big C++ project that I cannot split in multiple smaller projects? I have no choice but to create a big Python module exposing all the functions/classes? What is the common practice?

Regards,

Olivier

On Tue, Nov 15, 2011 at 9:24 AM, Jim Bosch <[hidden email]> wrote:

On Nov 15, 2011 8:53 AM, "Olivier Voyer" <[hidden email]> wrote:
>
> Hi everyone,
>
> Is it possible to have multiple modules linking to one single pyd file? I'm using SWIG with VS2010 and I can't find a way of doing that.
>

I believe this is not supported by the Python C-API itself, regardless of what wrapper generator or library you use.  The name of the module import function needs to be related to the loadable module file name, and you generally can't have two functions for which that's true.

Jim


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Wichert Akkerman
In reply to this post by Jérôme Laheurte
On 11/15/2011 03:51 PM, Jérôme Laheurte wrote:
> Le 15 nov. 2011 à 15:24, Jim Bosch a écrit :
>
>> On Nov 15, 2011 8:53 AM, "Olivier Voyer"<[hidden email]>  wrote:
>>> Hi everyone,
>>>
>>> Is it possible to have multiple modules linking to one single pyd file? I'm using SWIG with VS2010 and I can't find a way of doing that.
>>>
>> I believe this is not supported by the Python C-API itself, regardless of what wrapper generator or library you use.  The name of the module import function needs to be related to the loadable module file name, and you generally can't have two functions for which that's true.
> But you can put several submodules in an extension module, as a workaround (see code below). I always wondered if that was possible with boost::python or SWIG ?

You can. For example:


void exportModule1() {
     object module(handle<>(borrowed(PyImport_AddModule("mypkg.module1"))));
     scope().attr("module1") = module.
     scope module_scope = module;

    // put your def/class_/etc things here
}

void exportModule2() {
     object module(handle<>(borrowed(PyImport_AddModule("mypkg.module2"))));
     scope().attr("module2") = module.
     scope module_scope = module;

    // put your def/class_/etc things here
}

void export() {
     object package = scope();
     package.attr("__path__") = "mypkg";
     exportModule1();
     exportModule2();
}



_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Jim Bosch-2
In reply to this post by Olivier Voyer
On 11/15/2011 10:00 AM, Olivier Voyer wrote:
> Jim, thank you for your answer.
>
> What if I have this big C++ project that I cannot split in multiple
> smaller projects? I have no choice but to create a big Python module
> exposing all the functions/classes? What is the common practice?
>

It sounds like you just want to make a Python package - a directory with
an __init__.py file that contains all your submodules, which can each be
compiled separately.

Is there some reason you felt you needed to put all the modules in the
same pyd file?

Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Jim Bosch-2
In reply to this post by Jérôme Laheurte
On 11/15/2011 09:51 AM, Jérôme Laheurte wrote:
>
> But you can put several submodules in an extension module, as a
> workaround (see code below). I always wondered if that was possible
> with boost::python or SWIG ?

Interesting, I hadn't thought of that.

Anyhow, I believe it should be possible (but not very useful) with both;
you can make the inner module, but it's just a Python C-API module and
you have to manually add SWIG or Boost.Python-wrapped functions and
types to it.

Jim


>
>
> #include<Python.h>
>
> static PyObject *myfunc(PyObject *self, PyObject *args) { if
> (!PyArg_ParseTuple(args, "|myfunc")) return NULL;
>
> printf("Hello, world\n");
>
> Py_INCREF(Py_None); return Py_None; }
>
> static PyMethodDef functions[] = { { "myfunc", (PyCFunction)myfunc,
> METH_VARARGS, "" }, { NULL } };
>
> static PyMethodDef nofunctions[] = { { NULL } };
>
> void inittestmod() { PyObject *mdl, *submdl;
>
> if (!(mdl = Py_InitModule3("testmod", nofunctions, ""))) return;
>
> if (!(submdl = Py_InitModule3("foo", functions, ""))) return;
>
> Py_INCREF(submdl); PyModule_AddObject(mdl, "foo", submdl); }
>
> After building,
>
>>>> import testmod testmod.foo.myfunc()
> Hello, world
>>>>
>
> _______________________________________________ Cplusplus-sig mailing
> list [hidden email]
> http://mail.python.org/mailman/listinfo/cplusplus-sig

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Olivier Voyer
In reply to this post by Jim Bosch-2
Yes, that's exactly what I want to do. But, from what I understand, each module or submodule (.py file) must link to its own .pyd file, ie module1.py -> _module1.pyd, module2.py -> _module2.pyd.

What I would really love to have is:

myPackage/
myPackage/__init__.py
myPackage/module1.py -> linked to _mySingleAPI.pyd
myPackage/module2.py -> linked to _mySingleAPI.pyd

I just need a way to have some sort of modularity within my application. I cannot split my main project into smaller projects, it would require a huge amount of time.

On Tue, Nov 15, 2011 at 10:20 AM, Jim Bosch <[hidden email]> wrote:
On 11/15/2011 10:00 AM, Olivier Voyer wrote:
Jim, thank you for your answer.

What if I have this big C++ project that I cannot split in multiple
smaller projects? I have no choice but to create a big Python module
exposing all the functions/classes? What is the common practice?


It sounds like you just want to make a Python package - a directory with an __init__.py file that contains all your submodules, which can each be compiled separately.

Is there some reason you felt you needed to put all the modules in the same pyd file?


Jim
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Wichert Akkerman
On 11/15/2011 04:42 PM, Olivier Voyer wrote:

> Yes, that's exactly what I want to do. But, from what I understand,
> each module or submodule (.py file) must link to its own .pyd file, ie
> module1.py -> _module1.pyd, module2.py -> _module2.pyd.
>
> What I would really love to have is:
>
> myPackage/
> myPackage/__init__.py
> myPackage/module1.py -> linked to _mySingleAPI.pyd
> myPackage/module2.py -> linked to _mySingleAPI.pyd

Why not make mySingleAPI.pyd a single extension myPackage.pyd module
that creates myPackage, myPackage.module1, etc. all at once? You don't
need to have separate files for each module if you do that.

Wichert.

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Olivier Voyer
Bingo! But I'm not sure how to do that... at this moment I'm using SWIG as a C++ wrapper. Do I only need to modify the __init__py file at the root of my package?

On Tue, Nov 15, 2011 at 10:47 AM, Wichert Akkerman <[hidden email]> wrote:
On 11/15/2011 04:42 PM, Olivier Voyer wrote:
Yes, that's exactly what I want to do. But, from what I understand, each module or submodule (.py file) must link to its own .pyd file, ie module1.py -> _module1.pyd, module2.py -> _module2.pyd.

What I would really love to have is:

myPackage/
myPackage/__init__.py
myPackage/module1.py -> linked to _mySingleAPI.pyd
myPackage/module2.py -> linked to _mySingleAPI.pyd

Why not make mySingleAPI.pyd a single extension myPackage.pyd module that creates myPackage, myPackage.module1, etc. all at once? You don't need to have separate files for each module if you do that.

Wichert.


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig


_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Wichert Akkerman
On 11/15/2011 04:52 PM, Olivier Voyer wrote:
> Bingo! But I'm not sure how to do that... at this moment I'm using
> SWIG as a C++ wrapper. Do I only need to modify the __init__py file at
> the root of my package?

Personally I don't have a single .py file for my extensions, I just drop
in a single myPackage.so file (I don't use windows, but your equivalent
is probably a .pyd file) which exports all modules.

Wichert.
_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Jérôme Laheurte
In reply to this post by Jim Bosch-2

Le 15 nov. 2011 à 16:20, Jim Bosch a écrit :

> On 11/15/2011 10:00 AM, Olivier Voyer wrote:
>> Jim, thank you for your answer.
>>
>> What if I have this big C++ project that I cannot split in multiple
>> smaller projects? I have no choice but to create a big Python module
>> exposing all the functions/classes? What is the common practice?
>>
>
> It sounds like you just want to make a Python package - a directory with an __init__.py file that contains all your submodules, which can each be compiled separately.
>
> Is there some reason you felt you needed to put all the modules in the same pyd file?

To share common structures/classes without having to use a PyCObject. Actually that's rather theoretical, I never actually needed this, just curious.

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Jérôme Laheurte
In reply to this post by Wichert Akkerman

Le 15 nov. 2011 à 16:08, Wichert Akkerman a écrit :

> On 11/15/2011 03:51 PM, Jérôme Laheurte wrote:
>> Le 15 nov. 2011 à 15:24, Jim Bosch a écrit :
>>
>>> On Nov 15, 2011 8:53 AM, "Olivier Voyer"<[hidden email]>  wrote:
>>>> Hi everyone,
>>>>
>>>> Is it possible to have multiple modules linking to one single pyd file? I'm using SWIG with VS2010 and I can't find a way of doing that.
>>>>
>>> I believe this is not supported by the Python C-API itself, regardless of what wrapper generator or library you use.  The name of the module import function needs to be related to the loadable module file name, and you generally can't have two functions for which that's true.
>> But you can put several submodules in an extension module, as a workaround (see code below). I always wondered if that was possible with boost::python or SWIG ?
>
> You can. For example:
>
>
> void exportModule1() {
>    object module(handle<>(borrowed(PyImport_AddModule("mypkg.module1"))));
>    scope().attr("module1") = module.
>    scope module_scope = module;
>
>   // put your def/class_/etc things here
> }
>
> void exportModule2() {
>    object module(handle<>(borrowed(PyImport_AddModule("mypkg.module2"))));
>    scope().attr("module2") = module.
>    scope module_scope = module;
>
>   // put your def/class_/etc things here
> }
>
> void export() {
>    object package = scope();
>    package.attr("__path__") = "mypkg";
>    exportModule1();
>    exportModule2();
> }

Nice, thanks. I'll keep this around.

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Jérôme Laheurte
In reply to this post by Jérôme Laheurte

Le 15 nov. 2011 à 17:15, Jérôme Laheurte a écrit :

>
> Le 15 nov. 2011 à 16:20, Jim Bosch a écrit :
>
>> On 11/15/2011 10:00 AM, Olivier Voyer wrote:
>>> Jim, thank you for your answer.
>>>
>>> What if I have this big C++ project that I cannot split in multiple
>>> smaller projects? I have no choice but to create a big Python module
>>> exposing all the functions/classes? What is the common practice?
>>>
>>
>> It sounds like you just want to make a Python package - a directory with an __init__.py file that contains all your submodules, which can each be compiled separately.
>>
>> Is there some reason you felt you needed to put all the modules in the same pyd file?
>
> To share common structures/classes without having to use a PyCObject. Actually that's rather theoretical, I never actually needed this, just curious.

Nevermind, that's a silly reason. Of course I can just use the same classes from different .so files (as long as I rebuild them all when I change a declaration).

_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig
Reply | Threaded
Open this post in threaded view
|

Re: Multiple modules in a single pyd

Niall Douglas
In reply to this post by Olivier Voyer
On 15 Nov 2011 at 10:00, Olivier Voyer wrote:

> What if I have this big C++ project that I cannot split in multiple smaller
> projects? I have no choice but to create a big Python module exposing all
> the functions/classes? What is the common practice?

You are aware, I assume, that the python wrappings can live in a
separate DLL/SO than the thing you are wrapping?

So, you can have a monolithic big C++ project DLL/SO several dozen
megabytes in size, but with multiple wrapper DLL/SO's wrapping just a
portion of the APIs provided by the monolithic DLL/SO. Each can be
loaded, as needed, by the python runtime.

Generally speaking, one wants to try and keep symbol counts low when
possible. Dynamic linkers have become much better in recent years at
becoming O(N) with symbol count, but there still a few O(N^2)
behaviours in there. A very large DLL/SO therefore typically will be
much slower to load in than many smaller DLL/SOs even if they
represent the same amount of code. And on POSIX, don't forget to make
judicious use of "-fvisibility=hidden" and
"-fvisibility-inlines-hidden" as they can very dramatically reduce
symbol count for the linker.

HTH,
Niall

--
Technology & Consulting Services - ned Productions Limited.
http://www.nedproductions.biz/. VAT reg: IE 9708311Q. Company no:
472909.



_______________________________________________
Cplusplus-sig mailing list
[hidden email]
http://mail.python.org/mailman/listinfo/cplusplus-sig