Can boost.asio read/write io of files asynchronously?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
As title, I read boost.asio documents and generally said io about http/network.

I don't see any info. of io of files. Does boost.asio support read/write io of files asynchronously?

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
On 31/08/2020 14:59, lampahome wrote:
> As title, I read boost.asio documents and generally said io about
> http/network.
>
> I don't see any info. of io of files. Does boost.asio support read/write
> io of files asynchronously?

I haven't tried any of these methods, but it does provide the
posix::stream_descriptor and windows::{stream,random_access}_handle
classes which should be able to operate on native files.  (You'll have
to use a native OS-level function to open the handle/descriptor for the
file and pass ownership to the ASIO object; they lack means to do the
opening itself.)

https://www.boost.org/doc/libs/1_74_0/doc/html/boost_asio/overview/posix/stream_descriptor.html

https://www.boost.org/doc/libs/1_74_0/doc/html/boost_asio/overview/windows/stream_handle.html
https://www.boost.org/doc/libs/1_74_0/doc/html/boost_asio/overview/windows/random_access_handle.html


Alternatively, there's another library (that I also haven't used)
dedicated to async file I/O.  It originally started life based on ASIO
but I think it's since diverged into something else.  And called
something different from the last time I saw it as well.

Its current home appears to be: https://ned14.github.io/llfio/

(The impression I have, which may be mistaken, is that it is possibly
overkill for general application use, unless you're trying to implement
some kind of custom database engine rather than just using an
off-the-shelf DB library.)
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
I haven't tried any of these methods, but it does provide the
posix::stream_descriptor and windows::{stream,random_access}_handle
classes which should be able to operate on native files.  (You'll have
to use a native OS-level function to open the handle/descriptor for the
file and pass ownership to the ASIO object; they lack means to do the
opening itself.)

Thank u.
That gives me a direction. The only sad thing is very little document/examples described about this API of boost.asio .

 

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list


On Mon, 31 Aug 2020 at 15:01, lampahome via Boost-users <[hidden email]> wrote:
I haven't tried any of these methods, but it does provide the
posix::stream_descriptor and windows::{stream,random_access}_handle
classes which should be able to operate on native files.  (You'll have
to use a native OS-level function to open the handle/descriptor for the
file and pass ownership to the ASIO object; they lack means to do the
opening itself.)

Thank u.
That gives me a direction. The only sad thing is very little document/examples described about this API of boost.asio .

This is a common complaint of Asio.

If you would like to tap into a great body of knowledge and interest in asio/networking-ts you can do so by joining the #beast channel in cpplang slack 


Beast depends very heavily on Asio and we have worked to build a strong body of expertise.


 

 
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users


--
Richard Hodges
office: +442032898513
home: +376841522
mobile: +376380212


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
If you would like to tap into a great body of knowledge and interest in asio/networking-ts you can do so by joining the #beast channel in cpplang slack 


If I'm interest in asynchronous io of file, do I try asio?
What I look for is like libaio in unix system, but still found no wrapper/library to do like that.

thanks 

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
Asio certainly has support for asynchronous file IO. I have used it on linux systems for a multitude of file handle types, including console I/O.


On Tue, 1 Sep 2020 at 06:40, lampahome via Boost-users <[hidden email]> wrote:
If you would like to tap into a great body of knowledge and interest in asio/networking-ts you can do so by joining the #beast channel in cpplang slack 


If I'm interest in asynchronous io of file, do I try asio?
What I look for is like libaio in unix system, but still found no wrapper/library to do like that.

thanks 
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users


--
Richard Hodges
office: +442032898513
home: +376841522
mobile: +376380212


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
On 01/09/2020 11:54, Richard Hodges via Boost-users wrote:

> Asio certainly has support for asynchronous file IO. I have used it on
> linux systems for a multitude of file handle types, including console I/O.

Unless you were using a custom io_uring based i/o service, all your
asynchronous file i/o on Linux was actually synchronous all along.

Linux doesn't implement async file i/o until io_uring, and even then,
not properly until kernel 5.5 and even then again, not really until
kernel 5.9 releases. Some would argue that proper, reliable,
sans-pathological-corner-case async file i/o won't arrive in Linux until
some time in the 6.x kernel series.

Niall
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
In reply to this post by Boost - Users mailing list
On 31/08/2020 05:08, Gavin Lambert via Boost-users wrote:

> Alternatively, there's another library (that I also haven't used)
> dedicated to async file I/O.  It originally started life based on ASIO
> but I think it's since diverged into something else.  And called
> something different from the last time I saw it as well.
>
> Its current home appears to be: https://ned14.github.io/llfio/

Yes, one part of it should enter the C++ 23 standard, other parts may do so.

LLFIO doesn't export publicly any async file i/o implementation however.
This is because async file i/o usually has worse performance than
synchronous file i/o. So you generally want to use synchronous i/o
tightly integrated with kernel threads, if concurrency is desired (quite
often it is not).

> (The impression I have, which may be mistaken, is that it is possibly
> overkill for general application use, unless you're trying to implement
> some kind of custom database engine rather than just using an
> off-the-shelf DB library.)

Spot on. The only place where async i/o makes a lot of sense is where
i/o latencies are unpredictable most of the time. This is usually the
case with socket i/o latencies, it is rarely the case with file i/o (and
as you mention, only in very high random access use cases across
datasets far exceeding RAM does disabling caching for file i/o make sense).

Niall

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
In reply to this post by Boost - Users mailing list
Hello,

Niall Douglas Wrote:
> Unless you were using a custom io_uring based i/o service, all your asynchronous file i/o on Linux was actually synchronous all along.

It doesn't really matter, whether the IO really is performed asynchronously. The asio API is asynchronously in the sense, that the completion handler is not called from within the async_*() call but later from within the main loop and the main loop does not block, while waiting for the IO operation to complete.

It just happens to be the same Linux API, that is used for async socket IO and for async file io, since on linux systems, sockets are files.

But: Things probably are different on windows systems, so there is no guarantee.

73, Mario
--
Mario Klebsch                      Actia I+ME GmbH
[hidden email]         Dresdenstrasse 17/18
Fon: +49 531 38 701 716            38124 Braunschweig
Fax: +49 531 38 701 88            German

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
On 01/09/2020 15:30, Klebsch, Mario via Boost-users wrote:

> Niall Douglas Wrote:
>> Unless you were using a custom io_uring based i/o service, all your asynchronous file i/o on Linux was actually synchronous all along.
>
> It doesn't really matter, whether the IO really is performed asynchronously. The asio API is asynchronously in the sense, that the completion handler is not called from within the async_*() call but later from within the main loop and the main loop does not block, while waiting for the IO operation to complete.
>
> It just happens to be the same Linux API, that is used for async socket IO and for async file io, since on linux systems, sockets are files.

No, on Linux you cannot use epoll() with file descriptors which refer to
seekable devices. You get an error. As the ASIO i/o service on Linux
uses epoll(), one quite literally cannot multiplex i/o on files on Linux
with ASIO without a custom i/o service implementation.

If file i/o appears to work on ASIO on Linux, then ASIO is never handing
those file descriptors to epoll() at all, and is always blocking on
reads and writes. In this situation, it would be more efficient to call
the preadv() and pwritev() syscalls directly.

Niall
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list
In reply to this post by Boost - Users mailing list


Richard Hodges via Boost-users <[hidden email]> 於 2020年9月1日 週二 下午6:54寫道:
Asio certainly has support for asynchronous file IO. I have used it on linux systems for a multitude of file handle types, including console I/O.

 Is there any function can handle the finished IO operation?

Assume I have two threads. One is only submit async io to file, and another one is only handle event of those executed io.
I only find handler function but wonder if any easier handler to handle all.

thanks

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: Can boost.asio read/write io of files asynchronously?

Boost - Users mailing list


On Wed, 2 Sep 2020 at 06:59, lampahome <[hidden email]> wrote:


Richard Hodges via Boost-users <[hidden email]> 於 2020年9月1日 週二 下午6:54寫道:
Asio certainly has support for asynchronous file IO. I have used it on linux systems for a multitude of file handle types, including console I/O.

 Is there any function can handle the finished IO operation?

Assume I have two threads. One is only submit async io to file, and another one is only handle event of those executed io.
I only find handler function but wonder if any easier handler to handle all.

I'm not quite sure why you'd need two threads. But if you did, asio allows you to bind handlers to executors prior to submission to asynchronous initiating functions.
The handler will be executed by its associated executor, which could if you require be running in a different thread.

e.g.

async_write(f, buf, asio::bind_executor(other_executor, 
 [](error_code ec, std::size_t written) {
    // handler now executed by other_executor
 });

 If submitting across threads you'd want to be sure to protect access to the file object with a mutex or a strand.

 

thanks


--
Richard Hodges
office: +442032898513
home: +376841522
mobile: +376380212


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users