Memory mapped files

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Memory mapped files

Boost - Users mailing list
Good morning,

We are developing a system which heavily uses memory mapped files.
These files are initially created under a unix system, but will then be used in portable systems such as Android and iOS.
We are having some problems loading memory mapped files created under Unix into Android.

Is this scenario even possible ?
Are the memory mapped files compatible between systems ?

Hope you will be able to shed some light on this.
Best regards
Simon

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
On 21/01/2019 22:41, Simon Giddings wrote:
> We are developing a system which heavily uses memory mapped files.
> These files are initially created under a unix system, but will then be
> used in portable systems such as Android and iOS.
> We are having some problems loading memory mapped files created under
> Unix into Android.

In what context?  Using some Boost library or in general?

> Is this scenario even possible ?
> Are the memory mapped files compatible between systems ?

They can be, but are not guaranteed to be, depending on what you're
storing in the memory.

Binary layout of class types is usually not guaranteed cross-platform
(or cross-compiler, for that matter).

Anything involving pointers usually won't work at all, even
cross-process or cross-session, let alone cross-platform.

You can also have issues with padding or value endian ordering,
depending on the specific platforms involved.
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
In reply to this post by Boost - Users mailing list
> We are developing a system which heavily uses memory mapped files.
> These files are initially created under a unix system, but will then be
> used in portable systems such as Android and iOS.
> We are having some problems loading memory mapped files created under
> Unix into Android.
>
> Is this scenario even possible ?
> Are the memory mapped files compatible between systems ?

This kind of question has nothing to do with Boost, and would be better
asked on Stackoverflow.

However yes Android supports memory mapped files just fine. Free address
space is very limited though, so what works on 64 bit workstations may
not work on 32 bit Android. Also, Intel CPUs are not ARM CPUs, padding
of structs will be different, floating point numbers may have a
different representation or be interpreted a bit differently, there are
a few other differences.

With discipline memory mapped files usable on both architectures can be
devised, but usually with a performance penalty on all platforms. In the
end there is no free lunch.

Niall
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
Sorry, but this is a very specific Boost question as we are using the Boost memory mapping system.

So, if I understand correctly, you are saying that Boost does / cannot garantee compatibility between OS and cpu architecture ?
This would mean that Boost leverages heavily the underlying OS services where available.
Having said that, other than doing a "memory dump", is it possible to get Boost to serialise the data out to disk ?


On 23/01/2019 02:33, Niall Douglas via Boost-users wrote:
We are developing a system which heavily uses memory mapped files.
These files are initially created under a unix system, but will then be
used in portable systems such as Android and iOS.
We are having some problems loading memory mapped files created under
Unix into Android.

Is this scenario even possible ?
Are the memory mapped files compatible between systems ?
This kind of question has nothing to do with Boost, and would be better
asked on Stackoverflow.

However yes Android supports memory mapped files just fine. Free address
space is very limited though, so what works on 64 bit workstations may
not work on 32 bit Android. Also, Intel CPUs are not ARM CPUs, padding
of structs will be different, floating point numbers may have a
different representation or be interpreted a bit differently, there are
a few other differences.

With discipline memory mapped files usable on both architectures can be
devised, but usually with a performance penalty on all platforms. In the
end there is no free lunch.

Niall
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
In reply to this post by Boost - Users mailing list
We have already taken into account the problem of pointers and also endian ordering.
When doing a hex dump of the files produced under unix and one produced under android, using the same data content, we have noticed that there are byte differences.
So, other than doing a "memory dump", is it possible to get Boost to serialise the data out to disk ?

On 23/01/2019 01:46, Gavin Lambert via Boost-users wrote:
On 21/01/2019 22:41, Simon Giddings wrote:
We are developing a system which heavily uses memory mapped files.
These files are initially created under a unix system, but will then be used in portable systems such as Android and iOS.
We are having some problems loading memory mapped files created under Unix into Android.

In what context?  Using some Boost library or in general?

Is this scenario even possible ?
Are the memory mapped files compatible between systems ?

They can be, but are not guaranteed to be, depending on what you're storing in the memory.

Binary layout of class types is usually not guaranteed cross-platform (or cross-compiler, for that matter).

Anything involving pointers usually won't work at all, even cross-process or cross-session, let alone cross-platform.

You can also have issues with padding or value endian ordering, depending on the specific platforms involved.
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
In reply to this post by Boost - Users mailing list
On 1/23/19 3:15 AM, Simon Giddings via Boost-users wrote:
> Sorry, but this is a very specific Boost question as we are using the
> Boost memory mapping system.
>
> So, if I understand correctly, you are saying that Boost does / cannot
> garantee compatibility between OS and cpu architecture ?
> This would mean that Boost leverages heavily the underlying OS
> services where available.
> Having said that, other than doing a "memory dump", is it possible to
> get Boost to serialise the data out to disk ?

This is very much a generic language question, not a Boost question.

C++ make NO guarantee that in-memory organization of data structures
will match between different architectures, and there is nothing Boost
can do to make those structures match between architectures.

You can 'serialize' data in a architecture agnostic manner, but in
general, that will be nothing like a memory dump. To be totally general,
you need to dump the values in something like an ASCII representation.
If you can be sure that individual types have the same representation,
you can output that as a string of bytes for the individual types, and
reconstruct the types on reading. Do NOT expect class level object to be
streamed out at the byte level and be restore-able at the byte level, so
it will not be anything like a memory map file that can be just used as is.

--
Richard Damon

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
> C++ make NO guarantee that in-memory organization of data structures
> will match between different architectures, and there is nothing Boost
> can do to make those structures match between architectures.

It's even stronger than that: the C++ standard makes no guarantee that
in-memory representation of storage does not change between program
executions, let alone between different compilers or architectures. To
use mapped memory at all outside a single program execution is pure
undefined behaviour. You are literally on your own wrt the standard.

(I have a proposal in the works for WG21 SG12 which will propose how to
add support for shared and paged memory to C and C++, but even it
specifically excludes the ability to use storage by programs other than
the program which constructed the objects. And even that, the standard
requiring that the same program binary always uses the same in-memory
representation, will be very, very controversial e.g. imagine JITed or
translated C++ for example)

Niall
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
In reply to this post by Boost - Users mailing list
On 23/01/2019 21:15, Simon Giddings wrote:
> Sorry, but this is a very specific Boost question as we are using the
> Boost memory mapping system.

You still haven't indicated what library you're talking about.  There is
no "Boost memory mapping system"; there may or may not be a particular
library you are using.


But, as previously stated, C++ makes no particular guarantees about the
memory layout of class objects and only very minimal guarantees about
the layout of POD types.  This is especially true cross-platform and
cross-toolset, but can bite you when you update the compiler or runtimes
within a single platform, or even potentially from one run to another
(eg. hash/map ordering can change).

If you want portability, then you need to use a serialisation library
which implements portability (not all do) along with compatible data
structures; but this is an entirely different animal from plain memory
mapping.
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
In reply to this post by Boost - Users mailing list
On 1/23/19 12:19 AM, Simon Giddings via Boost-users wrote:
> We have already taken into account the problem of pointers and also
> endian ordering.
> When doing a hex dump of the files produced under unix and one produced
> under android, using the same data content, we have noticed that there
> are byte differences.
> So, other than doing a "memory dump", is it possible to get Boost to
> serialise the data out to disk ?
>

This is the purpose of the Boost Serialization library.  I'm quite sure
it (or likely any serialization library) would be quite suitable for for
addressing this problem.

Robert Ramey



_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
In reply to this post by Boost - Users mailing list
On 1/23/19 9:12 AM, Niall Douglas via Boost-users wrote:

>> C++ make NO guarantee that in-memory organization of data structures
>> will match between different architectures, and there is nothing Boost
>> can do to make those structures match between architectures.
> It's even stronger than that: the C++ standard makes no guarantee that
> in-memory representation of storage does not change between program
> executions, let alone between different compilers or architectures. To
> use mapped memory at all outside a single program execution is pure
> undefined behaviour. You are literally on your own wrt the standard.
>
> (I have a proposal in the works for WG21 SG12 which will propose how to
> add support for shared and paged memory to C and C++, but even it
> specifically excludes the ability to use storage by programs other than
> the program which constructed the objects. And even that, the standard
> requiring that the same program binary always uses the same in-memory
> representation, will be very, very controversial e.g. imagine JITed or
> translated C++ for example)
>
> Niall

While the standard doesn't explicitly define it, at least a POD which
doesn't contain pointers generally should maintain representation within
the same executable, even if run multiple times. The offsetof each of
the members is defined to be a compile time constant, so unless the
compiler uses some from of whole-program optimization to invoke the
as-if rule for structure layout, it can't be a run-time decision. The
representation of each of the fundamental members needs to be
documented, so unless the implementation documents a possible change of
representation at executable time, that will be consistent.

At a higher level, platform ABIs will generally define the
representation of structures, so that provides a stronger promise (and I
suspect all the implementations that Boost supports define themselves to
conforming to some platform ABI). This is needed for the linking
together of independently compiled modules.

Pointers will be an issue, as there is generally no assurance that the
program will run is the same memory space run to run (except on many
embedded platforms) so they would need to be fixed, and in general there
is no standard way to do this.

Non-PODs also have this issue, in part because they may include as part
of their representation pointers to related information (like the
v-table for virtual functions as an example).

--
Richard Damon

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Memory mapped files

Boost - Users mailing list
>> It's even stronger than that: the C++ standard makes no guarantee that
>> in-memory representation of storage does not change between program
>> executions, let alone between different compilers or architectures. To
>> use mapped memory at all outside a single program execution is pure
>> undefined behaviour. You are literally on your own wrt the standard.
>
> While the standard doesn't explicitly define it, at least a POD which
> doesn't contain pointers generally should maintain representation within
> the same executable, even if run multiple times.

It depends on your execution environment. An implementation based on an
interpreter or JIT may readily perform WPO which changes compile time
constants across program runs.

You may not think such execution environments common, but I'd remind you
of C++ on GPUs, or C++ cross compiled into Javascript.

Further on down the line we may have Modules from a Modules store be
late stage optimised into locally optimised binaries. Padding, in
particular, might be optimisable, especially as we start to support ever
larger integer types. Padding with 512-bit integers, for example, is
wasteful.

What I will agree with is that for a given version of a given toolchain
on a given architecture e.g. GCC 7.3 on x64, you currently get stronger
guarantees. But you shouldn't rely on anything not explicitly guaranteed
by the standard or your toolchain vendor.

Niall
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users