Is the async_read more reliable than async_read_some?

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
Hi,

I have been using asio::async_read with protobuf for some time, it is
relative easy to use protobuf function SerializeToOstream to attach a
packet header and send the packet header along with to protobuf
message, with fixed packet header size and message size in the header,
I was able to use async_read.

But I have to change to use msgpack which I could not find a way like
protobuf to attach a packet header to msgpack message. The only way I
could do is to use async_read_some, but I am very concerning the
remark in the async_read_some document. Appreciate your comments.

"The read operation may not read all of the requested number of bytes.
Consider using the async_read function if you need to ensure that the
requested amount of data is read before the asynchronous operation
completes."

Thank you.

- JH

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
AMDG

On 12/07/2018 02:44 AM, hh h via Boost wrote:

> I have been using asio::async_read with protobuf for some time, it is
> relative easy to use protobuf function SerializeToOstream to attach a
> packet header and send the packet header along with to protobuf
> message, with fixed packet header size and message size in the header,
> I was able to use async_read.
>
> But I have to change to use msgpack which I could not find a way like
> protobuf to attach a packet header to msgpack message. The only way I
> could do is to use async_read_some, but I am very concerning the
> remark in the async_read_some document. Appreciate your comments.
>

How do you identify the end of the message?
Can you use one of the overloads of async_read
that takes a completion_condition?

> "The read operation may not read all of the requested number of bytes.
> Consider using the async_read function if you need to ensure that the
> requested amount of data is read before the asynchronous operation
> completes."
>

aynyc_read is essentially implemented by calling
async_read_some repeatedly, until it fills
the requested size.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
> aynyc_read is essentially implemented by calling
> async_read_some repeatedly, until it fills
> the requested size.

That is exactly the question I am going to ask, if you don't know the
package size to read, you have to call sync_read_some with
max_buffer_size, how does the sync_read_some function know the end of
the message? Obviously, that the end of the message length is smaller
than the max_buffer_size. Is some timeout defined in sync_read_some to
identify the end of the message?

Thanks Steven

On 12/8/18, Steven Watanabe via Boost <[hidden email]> wrote:

> AMDG
>
> On 12/07/2018 02:44 AM, hh h via Boost wrote:
>> I have been using asio::async_read with protobuf for some time, it is
>> relative easy to use protobuf function SerializeToOstream to attach a
>> packet header and send the packet header along with to protobuf
>> message, with fixed packet header size and message size in the header,
>> I was able to use async_read.
>>
>> But I have to change to use msgpack which I could not find a way like
>> protobuf to attach a packet header to msgpack message. The only way I
>> could do is to use async_read_some, but I am very concerning the
>> remark in the async_read_some document. Appreciate your comments.
>>
>
> How do you identify the end of the message?
> Can you use one of the overloads of async_read
> that takes a completion_condition?
>
>> "The read operation may not read all of the requested number of bytes.
>> Consider using the async_read function if you need to ensure that the
>> requested amount of data is read before the asynchronous operation
>> completes."
>>
>
> aynyc_read is essentially implemented by calling
> async_read_some repeatedly, until it fills
> the requested size.
>
> In Christ,
> Steven Watanabe
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
AMDG

On 12/07/2018 05:05 PM, hh h wrote:

>> aynyc_read is essentially implemented by calling
>> async_read_some repeatedly, until it fills
>> the requested size.
>
> That is exactly the question I am going to ask, if you don't know the
> package size to read, you have to call sync_read_some with
> max_buffer_size, how does the sync_read_some function know the end of
> the message? Obviously, that the end of the message length is smaller
> than the max_buffer_size. Is some timeout defined in sync_read_some to
> identify the end of the message?
>

The way to identify the end of the message
depends on how the end of the message is
represented.  If it uses some kind of
delimiter, then async_read_until will
work.  If you signal the end of the message
by closing the connection, then you can
just read until EOF (which is indicated
by the error_code).

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
"If it uses some kind of delimiter, then async_read_until will work"

What kind of delimiter can I use to make async_read_until will work?

The server to call async_read_some will be connected by thousands
devices, can the async_read_some cope with thousands different
messages without causing problems?

I've just tried to run async_write using boost streambuf to pack
packet header and msgpack data together in one message data, but the
async_read in server site could only work a couple of minutes than it
crashed even I fixed all sizes in header file. It seems I can only use
the async_read_some for a stable server implementation.



On 12/8/18, Steven Watanabe via Boost <[hidden email]> wrote:

> AMDG
>
> On 12/07/2018 05:05 PM, hh h wrote:
>>> aynyc_read is essentially implemented by calling
>>> async_read_some repeatedly, until it fills
>>> the requested size.
>>
>> That is exactly the question I am going to ask, if you don't know the
>> package size to read, you have to call sync_read_some with
>> max_buffer_size, how does the sync_read_some function know the end of
>> the message? Obviously, that the end of the message length is smaller
>> than the max_buffer_size. Is some timeout defined in sync_read_some to
>> identify the end of the message?
>>
>
> The way to identify the end of the message
> depends on how the end of the message is
> represented.  If it uses some kind of
> delimiter, then async_read_until will
> work.  If you signal the end of the message
> by closing the connection, then you can
> just read until EOF (which is indicated
> by the error_code).
>
> In Christ,
> Steven Watanabe
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
AMDG

On 12/08/2018 04:51 PM, hh h wrote:
> "If it uses some kind of delimiter, then async_read_until will work"
>
> What kind of delimiter can I use to make async_read_until will work?
>

I have no idea.  If you don't know already then
your protocol is probably not designed to work
that way.  If you're designing your own protocol,
then sending the size first is probably simpler
to implement.

> The server to call async_read_some will be connected by thousands
> devices, can the async_read_some cope with thousands different
> messages without causing problems?
>
> I've just tried to run async_write using boost streambuf to pack
> packet header and msgpack data together in one message data, but the
> async_read in server site could only work a couple of minutes than it
> crashed even I fixed all sizes in header file. It seems I can only use
> the async_read_some for a stable server implementation.
>

How does it crash?  There's no fundamental reason
why prepending your own header with the size
wouldn't work.  The only likely issue that I
see with this setup is that async_read will receive
the entire message and buffer it in memory before it
signals completion, which might be a problem if
the messages are large.

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
>> What kind of delimiter can I use to make async_read_until will work?

>> The server to call async_read_some will be connected by thousands
>> devices, can the async_read_some cope with thousands different messages
>> without causing problems?

>> I've just tried to run async_write using boost streambuf to pack packet
>> header and msgpack data together in one message data, but the async_read
>> in server site could only work a couple of minutes than it crashed even I
>> fixed all sizes in header file. It seems I can only use the
>> async_read_some for a stable server implementation.

Describe your wire protocol, then we (those reading this message thread) can
give you some ideas. There are many ways to do the "message framing" (i.e.
determining the beginning and end of a message), but it all depends on how
the wire protocol is defined.

I assume you know that TCP / IP is a stream oriented protocol. There is no
direct correlation between how many bytes you send and how many (at one
time) will be received on the other end. That's why Asio provides many
different ways to consume the data on read.

Cliff




_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
> How does it crash?  There's no fundamental reason
> why prepending your own header with the size
> wouldn't work.

I sent 4 bytes header and 221 bytes messages repeatedly (every 2
seconds) calling from async_write of a sender, the receiver works
correctly and perfectly at async_read for a couple minutes, then it
crashed in the statement of readMessageBody for 221 bytes message at
async_read(socket_, boost::asio::buffer(buffer, size), ....), where
the size = 221n I don't think it is related to my program, either some
bugs in async_read or libssl. Here is my debug:


Read body size = 221

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770) at malloc.c:3123
3123    malloc.c: No such file or directory.

(gdb) backtrace
#0  0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770)
    at malloc.c:3123
#1  0x00007ffff796a8fc in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#2  0x00007ffff796a139 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#3  0x00007ffff796f9aa in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#4  0x00007ffff79797d5 in SSL_read ()
   from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
#5  0x00005555555665b1 in boost::asio::ssl::detail::engine::do_read (
    this=0x5555557d0fe0, data=0x5555557f7765, length=221)
    at /usr/include/boost/asio/ssl/detail/impl/engine.ipp:309

........

(gdb) print (char *)mem
$2 = 0x5555557f7770 "n\226"

Thanks Steven and Cliff.


On 12/9/18, Steven Watanabe via Boost <[hidden email]> wrote:

> AMDG
>
> On 12/08/2018 04:51 PM, hh h wrote:
>> "If it uses some kind of delimiter, then async_read_until will work"
>>
>> What kind of delimiter can I use to make async_read_until will work?
>>
>
> I have no idea.  If you don't know already then
> your protocol is probably not designed to work
> that way.  If you're designing your own protocol,
> then sending the size first is probably simpler
> to implement.
>
>> The server to call async_read_some will be connected by thousands
>> devices, can the async_read_some cope with thousands different
>> messages without causing problems?
>>
>> I've just tried to run async_write using boost streambuf to pack
>> packet header and msgpack data together in one message data, but the
>> async_read in server site could only work a couple of minutes than it
>> crashed even I fixed all sizes in header file. It seems I can only use
>> the async_read_some for a stable server implementation.
>>
>
> How does it crash?  There's no fundamental reason
> why prepending your own header with the size
> wouldn't work.  The only likely issue that I
> see with this setup is that async_read will receive
> the entire message and buffer it in memory before it
> signals completion, which might be a problem if
> the messages are large.
>
> In Christ,
> Steven Watanabe
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
AMDG

On 12/08/2018 06:50 PM, hh h wrote:

>> How does it crash?  There's no fundamental reason
>> why prepending your own header with the size
>> wouldn't work.
>
> I sent 4 bytes header and 221 bytes messages repeatedly (every 2
> seconds) calling from async_write of a sender, the receiver works
> correctly and perfectly at async_read for a couple minutes, then it
> crashed in the statement of readMessageBody for 221 bytes message at
> async_read(socket_, boost::asio::buffer(buffer, size), ....), where
> the size = 221n I don't think it is related to my program, either some
> bugs in async_read or libssl. Here is my debug:
>
>
> Read body size = 221
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770) at malloc.c:3123
> 3123    malloc.c: No such file or directory.
>
> (gdb) backtrace
> #0  0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770)
>     at malloc.c:3123
> #1  0x00007ffff796a8fc in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> #2  0x00007ffff796a139 in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> #3  0x00007ffff796f9aa in ?? () from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> #4  0x00007ffff79797d5 in SSL_read ()
>    from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> #5  0x00005555555665b1 in boost::asio::ssl::detail::engine::do_read (
>     this=0x5555557d0fe0, data=0x5555557f7765, length=221)
>     at /usr/include/boost/asio/ssl/detail/impl/engine.ipp:309
>
> ........
>
> (gdb) print (char *)mem
> $2 = 0x5555557f7770 "n\226"
>

This looks like some kind of memory corruption,
possibly a double free.  Have you tried valgrind
or addrsan?

In Christ,
Steven Watanabe

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Is the async_read more reliable than async_read_some?

Boost - Dev mailing list
Yes, I am doing the debug to further chasing it down...

Thanks Steven.



On 12/9/18, Steven Watanabe via Boost <[hidden email]> wrote:

> AMDG
>
> On 12/08/2018 06:50 PM, hh h wrote:
>>> How does it crash?  There's no fundamental reason
>>> why prepending your own header with the size
>>> wouldn't work.
>>
>> I sent 4 bytes header and 221 bytes messages repeatedly (every 2
>> seconds) calling from async_write of a sender, the receiver works
>> correctly and perfectly at async_read for a couple minutes, then it
>> crashed in the statement of readMessageBody for 221 bytes message at
>> async_read(socket_, boost::asio::buffer(buffer, size), ....), where
>> the size = 221n I don't think it is related to my program, either some
>> bugs in async_read or libssl. Here is my debug:
>>
>>
>> Read body size = 221
>>
>> Program received signal SIGSEGV, Segmentation fault.
>> 0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770) at
>> malloc.c:3123
>> 3123    malloc.c: No such file or directory.
>>
>> (gdb) backtrace
>> #0  0x00007ffff69d4c01 in __GI___libc_free (mem=0x5555557f7770)
>>     at malloc.c:3123
>> #1  0x00007ffff796a8fc in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> #2  0x00007ffff796a139 in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> #3  0x00007ffff796f9aa in ?? () from
>> /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> #4  0x00007ffff79797d5 in SSL_read ()
>>    from /usr/lib/x86_64-linux-gnu/libssl.so.1.1
>> #5  0x00005555555665b1 in boost::asio::ssl::detail::engine::do_read (
>>     this=0x5555557d0fe0, data=0x5555557f7765, length=221)
>>     at /usr/include/boost/asio/ssl/detail/impl/engine.ipp:309
>>
>> ........
>>
>> (gdb) print (char *)mem
>> $2 = 0x5555557f7770 "n\226"
>>
>
> This looks like some kind of memory corruption,
> possibly a double free.  Have you tried valgrind
> or addrsan?
>
> In Christ,
> Steven Watanabe
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost