Quantcast

Boost.HTTP ?

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Boost.HTTP ?

Michael Dickey
Greetings,

I've been working over the past 7+ months on a little library that  
uses Boost and ASIO to implement HTTP functionality.  Originally I  
called this "libpion," then renamed it to the "Pion Network Library."  
I'm using it as piece of a larger open source project for complex  
event processing, which has taken on the overall "Pion" brand (hence  
the change).

http://pion.atomiclabs.com/pion/net

The library includes pretty comprehensive support for HTTP versions  
1.0 and 1.1, including persistent connections, pipelining, chunked  
encodings, etc.  It supports client-side operations as well as an  
extensive server-side implementation, and can use either blocking or  
asynchronous sockets (through use of the Boost.ASIO library).

The original purpose was to create a Boost/C++ library for building  
lightweight HTTP interfaces for applications.  Although it can  
certainly be used to serve files, It is not meant to replace or  
compete with any full-featured web server (like Apache or lighttpd).  
One of its main features/uses is to bind HTTP resources to code.  The  
library is now fairly stable and feature-complete, and I have a few  
other developers helping out with it.  I've been thinking more lately  
about preparing this to submit for formal review as a Boost library.  
However, there are a few obstacles that I know of:

1) This is my first Boost library, and I made the mistake of using  
CamelCase and other non-Boost styles starting it out.  So, some re-
naming and cleanup work would obviously be required.  I don't imagine  
this would take too much time, though.

2) Some of the functionality overlaps with other libraries that have  
not yet been included or accepted into Boost (most notably, I think  
Boost.Extension would overlap with Pion's handling of web services:  
dynamic plug-ins bound to resources).  I'd be happy (and actually  
prefer) to swap out this code with Boost.Extension, but I do not  
believe it has been reviewed or accepted yet either.  I could also  
prepare a version with the plug-in functionality removed, but since  
it's being used in other projects, this would create a fork requiring  
extra maintenance energy, etc.

3) I know that Dean Michael Berris has formed a group of developers  
working on a more comprehensive network protocol library for Boost  
called cpp-netlib.  HTTP is one of the protocols they are working on.  
I've offered and would still be quite happy to combine efforts  
somehow, but that project still seems to have a long way to go, and a  
few people have mentioned that it may be preferable to have an  
independent library focused on HTTP.

So.. I decided to throw all this out there and see what people think.  
Should Boost have its own HTTP library, or should it be part of an  
more comprehensive network protocol library?  Would it be better have  
something available sooner in Boost that works and is reliable, and  
try to resolve the overlap over time as cpp-netlib matures?  Or, would  
it be better to wait and try to merge my library (or at least it's  
functionality) into cpp-netlib?

Thanks,
-Mike
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Jeff Garland
Michael Dickey wrote:

> Greetings,
>
> I've been working over the past 7+ months on a little library that  
> uses Boost and ASIO to implement HTTP functionality.  Originally I  
> called this "libpion," then renamed it to the "Pion Network Library."  
> I'm using it as piece of a larger open source project for complex  
> event processing, which has taken on the overall "Pion" brand (hence  
> the change).
>
> http://pion.atomiclabs.com/pion/net

Nice :-)

 >
 >...snip details...
 >
> So.. I decided to throw all this out there and see what people think.  
> Should Boost have its own HTTP library, or should it be part of an  
> more comprehensive network protocol library?  

Yes, it should have an HTTP library -- it would be nice if there were other
protocols, but not essential.

> Would it be better have  
> something available sooner in Boost that works and is reliable, and  
> try to resolve the overlap over time as cpp-netlib matures?  Or, would  
> it be better to wait and try to merge my library (or at least it's  
> functionality) into cpp-netlib?

I think we need a first protocol library to lay the groundwork.  Reviews and
interactions may lead to a set of standards for handling these sorts of
libraries.  If http lib is closer to ready then I'd say take it thru the
process as that might influence the design of cpp-netlib.  Of course, you can
take into account what the netlib folks are doing as well.  I'd hope that
somehow we could avoid redoing the http work...

Jeff

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Dean Michael Berris
In reply to this post by Michael Dickey
Hi Mike!

On Dec 10, 2007 9:29 AM, Michael Dickey <[hidden email]> wrote:
>
> 3) I know that Dean Michael Berris has formed a group of developers
> working on a more comprehensive network protocol library for Boost
> called cpp-netlib.  HTTP is one of the protocols they are working on.
> I've offered and would still be quite happy to combine efforts
> somehow, but that project still seems to have a long way to go, and a
> few people have mentioned that it may be preferable to have an
> independent library focused on HTTP.
>

You're right about working on a comprehensive network protocol
library, but there I think are a few differences with what libpion is
addressing and what cpp-netlib is trying to address. Let me try and
point these out as how I understand it (please correct me if I'm
wrong):

1. cpp-netlib primarily aims to implement a cross-platform,
header-only, standards compliant networking client library. It aims to
make using higher level application protocols easier than it currently
is on most platforms. The HTTP implementation is intended to primarily
be a client-side implementation, to make HTTP 1.0/1.1, (E)SMTP, among
other protocols available to client code with minimal intrusion : it
being a header-only library.

2. There has been initial intent to develop a server-side
implementation for HTTP in cpp-netlib which has been dropped because
there really is no single best way to implement and HTTP server --
because writing your own HTTP server usually means that you have
different trade-offs between performance and reliability not met by
those HTTP server implementations that are already out there.

3. Libpion aims to be in C++ what the Twisted framework is in Python:
a way to easily create HTTP-aware services using asynchronous
programming and non-blocking IO.

> So.. I decided to throw all this out there and see what people think.
> Should Boost have its own HTTP library, or should it be part of an
> more comprehensive network protocol library?  Would it be better have
> something available sooner in Boost that works and is reliable, and
> try to resolve the overlap over time as cpp-netlib matures?  Or, would
> it be better to wait and try to merge my library (or at least it's
> functionality) into cpp-netlib?
>

If you already have an HTTP 1.0 / 1.1 client implementation that is
licensed under the BSL (I see that libpion is already under the BSL)
and can be re-worked to use the basic_message implementation that's
already in cpp-netlib, then I think that would be a good start.

We can discuss this on either list (cpp-netlib-devel or this one), and
it would be something worth noting. If you look at the goals for the
1.0 release of cpp-netlib, it's to come up with a simple HTTP 1.0
client interface that's header-only, easy to use, and to come up with
the base on which most of the other network protocol implementations
will build upon. The message concept and implementation is already
available and is being currently extended to support a wide array of
policies/platforms (wide character support, chunking/linking, custom
allocators, custom string implementations, etc.).

Yes, it's going to be a lot of work, and I still need all the help I
can get -- with the day job taking most of the time and attention from
me, it's going to be hard to put in time to code all the stuff that's
still in my head wanting to get out someday. I plan on putting in a
lot more time during the holidays here in the Philippines (that's
between the 25th and the 1st of January) to implement more of the HTTP
operations and the unit tests that need to cover the implementations
that need to be written. So yes, it's years of man-hours of work, and
there's plenty for everyone who's interested. :D

I hope this helps!

--
Dean Michael C. Berris
Software Engineer, Friendster, Inc.
[http://cplusplus-soup.blogspot.com/]
[[hidden email]]
[+63 928 7291459]
[+1 408 4049523]
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Dean Michael Berris
In reply to this post by Jeff Garland
Hi Jeff!

On Dec 10, 2007 10:32 AM, Jeff Garland <[hidden email]> wrote:
> Michael Dickey wrote:
>  >
> > So.. I decided to throw all this out there and see what people think.
> > Should Boost have its own HTTP library, or should it be part of an
> > more comprehensive network protocol library?
>
> Yes, it should have an HTTP library -- it would be nice if there were other
> protocols, but not essential.
>

I have the same feeling, but then other protocols are becoming
increasingly more and more important as the web matures -- XMPP is
lurking to be the next generation IM/Messaging protocol, (E)SMTP is
not going away for Email anytime soon, and FTP is still very popular.
Maybe having a torrent client library might not be essential, though
if there's enough interest then it may just be the next generation
fail-safe P2P storage protocol -- or I might be dreaming too much. ;)

> > Would it be better have
> > something available sooner in Boost that works and is reliable, and
> > try to resolve the overlap over time as cpp-netlib matures?  Or, would
> > it be better to wait and try to merge my library (or at least it's
> > functionality) into cpp-netlib?
>
> I think we need a first protocol library to lay the groundwork.  Reviews and
> interactions may lead to a set of standards for handling these sorts of
> libraries.  If http lib is closer to ready then I'd say take it thru the
> process as that might influence the design of cpp-netlib.  Of course, you can
> take into account what the netlib folks are doing as well.  I'd hope that
> somehow we could avoid redoing the http work...
>

I agree.

If Mike already has an HTTP client library we can retro-fit to work
with the cpp-netlib basic_message<> implementation, then I think we
don't have to re-invent the wheel as far as an HTTP client
implementation goes -- and cpp-netlib 1.0 might just be around the
corner once we document it properly and get it tested up to Boost
standards.

Looking forward to hearing everyone's thoughts about this.

Have a good day everyone!

--
Dean Michael C. Berris
Software Engineer, Friendster, Inc.
[http://cplusplus-soup.blogspot.com/]
[[hidden email]]
[+63 928 7291459]
[+1 408 4049523]
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Jeff Garland
Dean Michael Berris wrote:

> Hi Jeff!
>
> On Dec 10, 2007 10:32 AM, Jeff Garland <[hidden email]> wrote:
>> Michael Dickey wrote:
>>  >
>>> So.. I decided to throw all this out there and see what people think.
>>> Should Boost have its own HTTP library, or should it be part of an
>>> more comprehensive network protocol library?
>> Yes, it should have an HTTP library -- it would be nice if there were other
>> protocols, but not essential.
>>
>
> I have the same feeling, but then other protocols are becoming
> increasingly more and more important as the web matures -- XMPP is
> lurking to be the next generation IM/Messaging protocol, (E)SMTP is
> not going away for Email anytime soon, and FTP is still very popular.
> Maybe having a torrent client library might not be essential, though
> if there's enough interest then it may just be the next generation
> fail-safe P2P storage protocol -- or I might be dreaming too much. ;)

I didn't mean to 'dis' the importance of the other protocols.  What I meant to
say is that if we try to bring an entire suite as one library, in one review,
it will a) take a long time, and b) be hard to manage.  So I'd rather see them
come as smaller contributions -- perhaps within a shared framework boost::net
or whatever.

>>> Would it be better have
>>> something available sooner in Boost that works and is reliable, and
>>> try to resolve the overlap over time as cpp-netlib matures?  Or, would
>>> it be better to wait and try to merge my library (or at least it's
>>> functionality) into cpp-netlib?
>> I think we need a first protocol library to lay the groundwork.  Reviews and
>> interactions may lead to a set of standards for handling these sorts of
>> libraries.  If http lib is closer to ready then I'd say take it thru the
>> process as that might influence the design of cpp-netlib.  Of course, you can
>> take into account what the netlib folks are doing as well.  I'd hope that
>> somehow we could avoid redoing the http work...
>>
>
> I agree.
>
> If Mike already has an HTTP client library we can retro-fit to work
> with the cpp-netlib basic_message<> implementation, then I think we
> don't have to re-invent the wheel as far as an HTTP client
> implementation goes -- and cpp-netlib 1.0 might just be around the
> corner once we document it properly and get it tested up to Boost
> standards.

Looks to me like Mike is focused on the server side...so maybe there's not
much overlap anyway.

Jeff
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Dean Michael Berris
On Dec 10, 2007 10:43 AM, Jeff Garland <[hidden email]> wrote:

> Dean Michael Berris wrote:
> > Hi Jeff!
> >
> > On Dec 10, 2007 10:32 AM, Jeff Garland <[hidden email]> wrote:
> >> Michael Dickey wrote:
> >>  >
> >>> So.. I decided to throw all this out there and see what people think.
> >>> Should Boost have its own HTTP library, or should it be part of an
> >>> more comprehensive network protocol library?
> >> Yes, it should have an HTTP library -- it would be nice if there were other
> >> protocols, but not essential.
> >>
> >
> > I have the same feeling, but then other protocols are becoming
> > increasingly more and more important as the web matures -- XMPP is
> > lurking to be the next generation IM/Messaging protocol, (E)SMTP is
> > not going away for Email anytime soon, and FTP is still very popular.
> > Maybe having a torrent client library might not be essential, though
> > if there's enough interest then it may just be the next generation
> > fail-safe P2P storage protocol -- or I might be dreaming too much. ;)
>
> I didn't mean to 'dis' the importance of the other protocols.  What I meant to
> say is that if we try to bring an entire suite as one library, in one review,
> it will a) take a long time, and b) be hard to manage.  So I'd rather see them
> come as smaller contributions -- perhaps within a shared framework boost::net
> or whatever.
>

Ah, yes. Now that makes sense to me.

As to doing it by piece meal (HTTP first, then SMTP next perhaps)
maybe we'd get more mileage. :)

> >
> > If Mike already has an HTTP client library we can retro-fit to work
> > with the cpp-netlib basic_message<> implementation, then I think we
> > don't have to re-invent the wheel as far as an HTTP client
> > implementation goes -- and cpp-netlib 1.0 might just be around the
> > corner once we document it properly and get it tested up to Boost
> > standards.
>
> Looks to me like Mike is focused on the server side...so maybe there's not
> much overlap anyway.
>

Too bad... That doesn't change though, cpp-netlib will be focused on
the client side. Insights from the libpion implementation may help
though, especially with HTTP 1.0/1.1 implementation details.

--
Dean Michael C. Berris
Software Engineer, Friendster, Inc.
[http://cplusplus-soup.blogspot.com/]
[[hidden email]]
[+63 928 7291459]
[+1 408 4049523]
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Chris Fairles
I found pion the other day, I used the scheduler and connection idea
(stripped down, non-camel case) for an irc client lib (in the works).

You can find the code here: http://skotty.coffeebuzzin.com/browser/trunk

I've been looking for some help and guidance on making a "boost.irc"
similar to this http (and boost.net) idea. Currently the message
parser relies on boost.spirit (implements the EBNF almost verbatim
from the RFC) and I only have client-components implemented. Server
will be next.

I also have some FTP stuff: http://sourceforge.net/projects/bftp but
haven't had time to make it complete and generic. It's just a basic
client again, no server components.

I'll try to keep up to date on this, quite busy but maybe some of this
code can be of some use.

Cheers,
Chris

On Dec 9, 2007 10:09 PM, Dean Michael Berris <[hidden email]> wrote:

> On Dec 10, 2007 10:43 AM, Jeff Garland <[hidden email]> wrote:
> > Dean Michael Berris wrote:
> > > Hi Jeff!
> > >
> > > On Dec 10, 2007 10:32 AM, Jeff Garland <[hidden email]> wrote:
> > >> Michael Dickey wrote:
> > >>  >
> > >>> So.. I decided to throw all this out there and see what people think.
> > >>> Should Boost have its own HTTP library, or should it be part of an
> > >>> more comprehensive network protocol library?
> > >> Yes, it should have an HTTP library -- it would be nice if there were other
> > >> protocols, but not essential.
> > >>
> > >
> > > I have the same feeling, but then other protocols are becoming
> > > increasingly more and more important as the web matures -- XMPP is
> > > lurking to be the next generation IM/Messaging protocol, (E)SMTP is
> > > not going away for Email anytime soon, and FTP is still very popular.
> > > Maybe having a torrent client library might not be essential, though
> > > if there's enough interest then it may just be the next generation
> > > fail-safe P2P storage protocol -- or I might be dreaming too much. ;)
> >
> > I didn't mean to 'dis' the importance of the other protocols.  What I meant to
> > say is that if we try to bring an entire suite as one library, in one review,
> > it will a) take a long time, and b) be hard to manage.  So I'd rather see them
> > come as smaller contributions -- perhaps within a shared framework boost::net
> > or whatever.
> >
>
> Ah, yes. Now that makes sense to me.
>
> As to doing it by piece meal (HTTP first, then SMTP next perhaps)
> maybe we'd get more mileage. :)
>
> > >
> > > If Mike already has an HTTP client library we can retro-fit to work
> > > with the cpp-netlib basic_message<> implementation, then I think we
> > > don't have to re-invent the wheel as far as an HTTP client
> > > implementation goes -- and cpp-netlib 1.0 might just be around the
> > > corner once we document it properly and get it tested up to Boost
> > > standards.
> >
> > Looks to me like Mike is focused on the server side...so maybe there's not
> > much overlap anyway.
> >
>
> Too bad... That doesn't change though, cpp-netlib will be focused on
> the client side. Insights from the libpion implementation may help
> though, especially with HTTP 1.0/1.1 implementation details.
>
> --
> Dean Michael C. Berris
> Software Engineer, Friendster, Inc.
> [http://cplusplus-soup.blogspot.com/]
> [[hidden email]]
> [+63 928 7291459]
> [+1 408 4049523]
> _______________________________________________
>
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Phil Endecott-48
In reply to this post by Dean Michael Berris
Dean Michael Berris wrote:
> 2. There has been initial intent to develop a server-side
> implementation for HTTP in cpp-netlib which has been dropped because
> there really is no single best way to implement an HTTP server

Yes, absolutely.  I think that the most useful approach would be to
provide building-blocks for things like HTTP header and URI parsing [I
have some Spirit code for this that I'd be prepared to contribute],
compression [Sebastian Redl seems to have gzip in his IOChains
proposal, but can it be re-used in an HTTP server without bringing lots
of baggage?], encryption, authentication, ETags (essentially hashes of
the content) and conditional fetches -- all of this done with attention
to security.  This really needs to be done in a way that will fit in to
thread-per-connection, thread pool, select()-based and other server
designs, and perhaps also as a CGI program or Apache server module.

How far can the server implementation be decoupled from the
content-specific stuff, and what interfaces should there be between them?

Michael Dickey wrote:
>> would it be better to wait and try to merge my library (or at least its
>> functionality) into cpp-netlib?

Far from merging your library into something else, I encourage you to
see how much you can break it up into smaller chunks and to make it
compatible with, yet not dependent on, other libraries.


Remember that there's also some GSoC CGI code pending.


Regards,  Phil.




_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Darren Garvey-3
On 10/12/2007, Phil Endecott <[hidden email]> wrote:
>
> Far from merging your library into something else, I encourage you to
> see how much you can break it up into smaller chunks and to make it
> compatible with, yet not dependent on, other libraries.
>
>
> Remember that there's also some GSoC CGI code pending.


Indeed there is! There's still a few months work left in it,
unfortunately... I think libpion takes a different approach than the CGI
library (there has been some recent discussion on the asio-users list about
this). I've been keeping in mind the cpp-netlib project too so hopefully
some form of protocol-independent basic_message<> class can be integrated to
any CGI library that gets into Boost.

It'd be ideal if a network building-blocks/utility library was accepted into
Boost, but at the same time there seems to be a lot of interest for
domain-specific libraries: like HTTP/IRC/CGI/etc. Hopefully these can all be
built on shared generic network utilities (that exist just-above Boost.Asio
).

As for which should come first, I have no idea. Either way, there is a lot
of overlap between CGI and HTTP, so there is a good chance quite a lot of
code can be shared between any two implementations. I hope so, anyway.

Regards,
Darren
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Michael Dickey
In reply to this post by Dean Michael Berris
On Dec 9, 2007, at 6:33 PM, Dean Michael Berris wrote:

> Hi Mike!
>
> On Dec 10, 2007 9:29 AM, Michael Dickey <[hidden email]> wrote:
>>
>> 3) I know that Dean Michael Berris has formed a group of developers
>> working on a more comprehensive network protocol library for Boost
>> called cpp-netlib.  HTTP is one of the protocols they are working on.
>> I've offered and would still be quite happy to combine efforts
>> somehow, but that project still seems to have a long way to go, and a
>> few people have mentioned that it may be preferable to have an
>> independent library focused on HTTP.
>>
>
> You're right about working on a comprehensive network protocol
> library, but there I think are a few differences with what libpion is
> addressing and what cpp-netlib is trying to address. Let me try and
> point these out as how I understand it (please correct me if I'm
> wrong):
>
> 1. cpp-netlib primarily aims to implement a cross-platform,
> header-only, standards compliant networking client library. It aims to
> make using higher level application protocols easier than it currently
> is on most platforms. The HTTP implementation is intended to primarily
> be a client-side implementation, to make HTTP 1.0/1.1, (E)SMTP, among
> other protocols available to client code with minimal intrusion : it
> being a header-only library.
>
> 2. There has been initial intent to develop a server-side
> implementation for HTTP in cpp-netlib which has been dropped because
> there really is no single best way to implement and HTTP server --
> because writing your own HTTP server usually means that you have
> different trade-offs between performance and reliability not met by
> those HTTP server implementations that are already out there.
>
> 3. Libpion aims to be in C++ what the Twisted framework is in Python:
> a way to easily create HTTP-aware services using asynchronous
> programming and non-blocking IO.


Sounds right to me.  I didn't realize cpp-netlib had decided to drop  
server-side support.  I think it makes sense though, since I agree  
that the two are very differently problems.

I started work on libpion with the server-side approach only in mind,  
and that is definitely its focus and strength.  However, in the 0.4.0  
(most recent) release, we did add support for client-side HTTP as  
well.  Mainly, because we wanted something better to use for our unit  
testing, and I realized that once you have a reusable HTTP parser  
(plus ASIO), it was trivial to add client-side support.

>> So.. I decided to throw all this out there and see what people think.
>> Should Boost have its own HTTP library, or should it be part of an
>> more comprehensive network protocol library?  Would it be better have
>> something available sooner in Boost that works and is reliable, and
>> try to resolve the overlap over time as cpp-netlib matures?  Or,  
>> would
>> it be better to wait and try to merge my library (or at least it's
>> functionality) into cpp-netlib?
>>
>
> If you already have an HTTP 1.0 / 1.1 client implementation that is
> licensed under the BSL (I see that libpion is already under the BSL)
> and can be re-worked to use the basic_message implementation that's
> already in cpp-netlib, then I think that would be a good start.

I agree that this sounds like a good place to start.  In particular,  
with the latest release we really cleaned-up and broke out the HTTP  
parsing code (which I should say was largely based on the code in  
Christopher's ASIO http_server examples) so that it can be used for  
either requests or responses, and can even be used without networking  
code (our intention is to use it also with an HTTP sniffer).

Your basic_message design is certainly cleaner and more extensible  
than our own.  We basically just have HTTPRequest and HTTPResponse  
classes that extend a base HTTPMessage class (with headers, version,  
etc).  We should be able to rework the parser so that it uses your  
classes instead, and be able to share that code between the two  
projects.

> We can discuss this on either list (cpp-netlib-devel or this one), and
> it would be something worth noting. If you look at the goals for the
> 1.0 release of cpp-netlib, it's to come up with a simple HTTP 1.0
> client interface that's header-only, easy to use, and to come up with
> the base on which most of the other network protocol implementations
> will build upon. The message concept and implementation is already
> available and is being currently extended to support a wide array of
> policies/platforms (wide character support, chunking/linking, custom
> allocators, custom string implementations, etc.).
>
> Yes, it's going to be a lot of work, and I still need all the help I
> can get -- with the day job taking most of the time and attention from
> me, it's going to be hard to put in time to code all the stuff that's
> still in my head wanting to get out someday. I plan on putting in a
> lot more time during the holidays here in the Philippines (that's
> between the 25th and the 1st of January) to implement more of the HTTP
> operations and the unit tests that need to cover the implementations
> that need to be written. So yes, it's years of man-hours of work, and
> there's plenty for everyone who's interested. :D

Unfortunately, I'm really pressed for time as well and will probably  
not be able to do much over the next couple months.  Might be able to  
find some time though to get the ball rolling.

Take care,
-Mike

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Michael Dickey
In reply to this post by Dean Michael Berris
Dean,

I think we could combine cpp-netlib's messaging classes with pion's  
HTTP parsing classes to establish a basis for HTTP processing and  
provide client-side support for HTTP/1.0 and 1.1.

There's two possible approaches to this that I see, one is to merge a  
"server" space into the Network library:

boost::net::basic_message<>
boost::net::http::request : public basic_message<>
boost::net::http::response : public basic_message<>
boost::net::server::tcp
boost::net::server::http : public tcp

Another would be to create an independent "server" library that uses  
the Network library classes:

boost::server::tcp
boost::server::http : public tcp

They both seem like sensible approaches to me.  In either case, I  
think that HTTP would be a good starting focus for both, to get a  
working foundation incorporated into boost.  We could always submit  
additional protocols independently for review as their implementations  
are finished.

What do you think?

Take care,
-Mike


On Dec 9, 2007, at 7:09 PM, Dean Michael Berris wrote:

>> I didn't mean to 'dis' the importance of the other protocols.  What  
>> I meant to
>> say is that if we try to bring an entire suite as one library, in  
>> one review,
>> it will a) take a long time, and b) be hard to manage.  So I'd  
>> rather see them
>> come as smaller contributions -- perhaps within a shared framework  
>> boost::net
>> or whatever.
>>
>
> Ah, yes. Now that makes sense to me.
>
> As to doing it by piece meal (HTTP first, then SMTP next perhaps)
> maybe we'd get more mileage. :)
>>>
>>> If Mike already has an HTTP client library we can retro-fit to work
>>> with the cpp-netlib basic_message<> implementation, then I think we
>>> don't have to re-invent the wheel as far as an HTTP client
>>> implementation goes -- and cpp-netlib 1.0 might just be around the
>>> corner once we document it properly and get it tested up to Boost
>>> standards.
>>
>> Looks to me like Mike is focused on the server side...so maybe  
>> there's not
>> much overlap anyway.
>>
>
> Too bad... That doesn't change though, cpp-netlib will be focused on
> the client side. Insights from the libpion implementation may help
> though, especially with HTTP 1.0/1.1 implementation details.
>
> --
> Dean Michael C. Berris
> Software Engineer, Friendster, Inc.
> [http://cplusplus-soup.blogspot.com/]
> [[hidden email]]
> [+63 928 7291459]
> [+1 408 4049523]
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Glyn Matthews
Mike, Dean,

On 10/12/2007, Michael Dickey <[hidden email]> wrote:
>
> Dean,
>
> I think we could combine cpp-netlib's messaging classes with pion's
> HTTP parsing classes to establish a basis for HTTP processing and
> provide client-side support for HTTP/1.0 and 1.1.



FWIW, I'm familiar with both libpion and cpp-netlib and I would agree that a
crossover here would be of benefit to both groups.


> They both seem like sensible approaches to me.  In either case, I
> think that HTTP would be a good starting focus for both, to get a
> working foundation incorporated into boost.  We could always submit
> additional protocols independently for review as their implementations
> are finished.


I completely agree with this too.  I think there are other individuals who
want to work on other protocols using cpp-netlib, but at different speeds
and more independently.  I think this is the best way to make progress.

Glyn
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Michael Dickey
In reply to this post by Phil Endecott-48
On Dec 10, 2007, at 6:39 AM, Phil Endecott wrote:

> Dean Michael Berris wrote:
>> 2. There has been initial intent to develop a server-side
>> implementation for HTTP in cpp-netlib which has been dropped because
>> there really is no single best way to implement an HTTP server
>
> Yes, absolutely.  I think that the most useful approach would be to
> provide building-blocks for things like HTTP header and URI parsing [I
> have some Spirit code for this that I'd be prepared to contribute],
> compression [Sebastian Redl seems to have gzip in his IOChains
> proposal, but can it be re-used in an HTTP server without bringing  
> lots
> of baggage?], encryption, authentication, ETags (essentially hashes of
> the content) and conditional fetches -- all of this done with  
> attention
> to security.  This really needs to be done in a way that will fit in  
> to
> thread-per-connection, thread pool, select()-based and other server
> designs, and perhaps also as a CGI program or Apache server module.
>
> How far can the server implementation be decoupled from the
> content-specific stuff, and what interfaces should there be between  
> them?
>
> Michael Dickey wrote:
>>> would it be better to wait and try to merge my library (or at  
>>> least its
>>> functionality) into cpp-netlib?
>
> Far from merging your library into something else, I encourage you to
> see how much you can break it up into smaller chunks and to make it
> compatible with, yet not dependent on, other libraries.

It's definitely moving in that direction.  For example, you can use  
the client-side part of the library without touching any server-side  
stuff, and you can even parse HTTP messages without any networking  
code.  I agree with the approach of breaking pieces out and pushing  
them into a lower-level space like cpp-netlib  over time.

I guess a big question is, does _server-side_ functionality belong in  
Boost?  I think for most cases, you're going to use an external  
packaged server to solve a problem.  However, I think HTTP is a unique  
case in that it is so widely used and for such a wide variety of  
applications, and is simple enough that it often is embedded within  
applications.

> Remember that there's also some GSoC CGI code pending.

Where is that project located?

Take care,
-Mike
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Andrey Tcherepanov
In reply to this post by Michael Dickey
On Mon, 10 Dec 2007 13:28:42 -0700, Michael Dickey <[hidden email]>  
wrote:

[snip]
> There's two possible approaches to this that I see, one is to merge a
> "server" space into the Network library:
>
> boost::net::basic_message<>
> boost::net::http::request : public basic_message<>
> boost::net::http::response : public basic_message<>
> boost::net::server::tcp

> boost::net::server::http : public tcp

> Another would be to create an independent "server" library that uses
> the Network library classes:
>
> boost::server::tcp
> boost::server::http : public tcp
>
> They both seem like sensible approaches to me.  In either case, I
> think that HTTP would be a good starting focus for both, to get a
> working foundation incorporated into boost.  We could always submit
> additional protocols independently for review as their implementations
> are finished.
>
> What do you think?
[snip]

(assuming that "what you think was directed to all of us, not just Dean  
:), so below is my $0.02 CAD)

I like the idea of basic_message<> (which could be used for SMTP and gods  
know what else as well), but what about UDP servers or, generally  
speaking, multi-transport protocols?

Although http is sitting on top of TCP, there is an RTSP protocol which  
claims to be derived from HTTP/1.1 in most parts (well, at least RFC has  
lots of backrefs to HTTP spec), but allows to use connectionless transport  
(aka UDP). Moreover, RTSP treats Clients as something that can receive  
asynchronous stream change notifications (if allowed). I am not arguing  
that your design will not cover 95% of useful cases, but I think protocol  
parsing and transport layer might need more separation. This way if brave  
person will decide to implement RTSP (for example) as derived from HTTP,  
he/she will not be limited by TCP transport.

Thanks,
   Andrey

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Michael Dickey
On Dec 10, 2007, at 1:00 PM, Andrey Tcherepanov wrote:

>> ...
>> What do you think?
> [snip]
>
> (assuming that "what you think was directed to all of us, not just  
> Dean
> :), so below is my $0.02 CAD)

Of course, otherwise I wouldn't be filling you INBOX with all these  
messages =)

> I like the idea of basic_message<> (which could be used for SMTP and  
> gods
> know what else as well), but what about UDP servers or, generally
> speaking, multi-transport protocols?
>
> Although http is sitting on top of TCP, there is an RTSP protocol  
> which
> claims to be derived from HTTP/1.1 in most parts (well, at least RFC  
> has
> lots of backrefs to HTTP spec), but allows to use connectionless  
> transport
> (aka UDP). Moreover, RTSP treats Clients as something that can receive
> asynchronous stream change notifications (if allowed). I am not  
> arguing
> that your design will not cover 95% of useful cases, but I think  
> protocol
> parsing and transport layer might need more separation. This way if  
> brave
> person will decide to implement RTSP (for example) as derived from  
> HTTP,
> he/she will not be limited by TCP transport.

In the current Pion code, the parsing & transport are handled  
independently.  I've never heard of HTTP being used on top of  
something other than TCP.  But I think we could probably accommodate  
for this by slightly altering the tcp_server / http_server class  
hierarchy.

Take care,
-Mike
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Phil Endecott-48
In reply to this post by Michael Dickey
Michael Dickey wrote:

> On Dec 10, 2007, at 6:39 AM, Phil Endecott wrote:
>> Far from merging your library into something else, I encourage you to
>> see how much you can break it up into smaller chunks and to make it
>> compatible with, yet not dependent on, other libraries.
>
> It's definitely moving in that direction.  For example, you can use  
> the client-side part of the library without touching any server-side  
> stuff, and you can even parse HTTP messages without any networking  
> code.  I agree with the approach of breaking pieces out and pushing  
> them into a lower-level space like cpp-netlib  over time.
>
> I guess a big question is, does _server-side_ functionality belong in  
> Boost?  I think for most cases, you're going to use an external  
> packaged server to solve a problem.

I might have agreed with you a few years ago but recently I've come to
the conclusion that it's better to just implement an HTTP server.  In
the past I have written CGI programs, which have the problem of
starting one process per request, and Apache modules, where I have used
Boost.Interprocess to store inter-request data.  Just writing an HTTP
server is simpler and better, and you can get Apache features
(compression, encryption, authentication, etc. etc.) by using its
mod_proxy to forward requests to your separate server.


Phil.




_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Michael Dickey
On Dec 10, 2007, at 2:12 PM, Phil Endecott wrote:

> Michael Dickey wrote:
>> On Dec 10, 2007, at 6:39 AM, Phil Endecott wrote:
>>> Far from merging your library into something else, I encourage you  
>>> to
>>> see how much you can break it up into smaller chunks and to make it
>>> compatible with, yet not dependent on, other libraries.
>>
>> It's definitely moving in that direction.  For example, you can use
>> the client-side part of the library without touching any server-side
>> stuff, and you can even parse HTTP messages without any networking
>> code.  I agree with the approach of breaking pieces out and pushing
>> them into a lower-level space like cpp-netlib  over time.
>>
>> I guess a big question is, does _server-side_ functionality belong in
>> Boost?  I think for most cases, you're going to use an external
>> packaged server to solve a problem.
>
> I might have agreed with you a few years ago but recently I've come to
> the conclusion that it's better to just implement an HTTP server.  In
> the past I have written CGI programs, which have the problem of
> starting one process per request, and Apache modules, where I have  
> used
> Boost.Interprocess to store inter-request data.  Just writing an HTTP
> server is simpler and better, and you can get Apache features
> (compression, encryption, authentication, etc. etc.) by using its
> mod_proxy to forward requests to your separate server.

I agree; that's why I wrote the library: to make it really easy to  
embed your own HTTP server =)

I think what I was trying to say is just that for most servers, taking  
an embedded/roll-your-own path is hard to justify.  But HTTP is  
becoming somewhat of a special case, in that it's more often easier to  
justify embedding a simple server in your application, versus  
integration with an outside server using CGI or some other API.

Take care,
-Mike
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Dean Michael Berris
In reply to this post by Michael Dickey
Hi Mike!

On Dec 11, 2007 3:54 AM, Michael Dickey <[hidden email]> wrote:
>
> On Dec 9, 2007, at 6:33 PM, Dean Michael Berris wrote:
>
>
> Sounds right to me.  I didn't realize cpp-netlib had decided to drop
> server-side support.  I think it makes sense though, since I agree
> that the two are very differently problems.
>

Yeah, it was some sort of consensus that pretty much said -- there
were too many ways to do server side HTTP, that's not going to be
really practical to try to cover the different tradeoffs between
performance, reliability, and scalability in a library. Plus, with the
aim of doing it 'header-only' style, the amount of effort to be put
into it doesn't make much sense (yet).

I don't mind working with libpion myself, but maybe not in the
projects I'm currently involved in. :)

> I started work on libpion with the server-side approach only in mind,
> and that is definitely its focus and strength.  However, in the 0.4.0
> (most recent) release, we did add support for client-side HTTP as
> well.  Mainly, because we wanted something better to use for our unit
> testing, and I realized that once you have a reusable HTTP parser
> (plus ASIO), it was trivial to add client-side support.
>

I'll try to take a look at the client code and see what insights
cpp-netlib may gain especially with regards to the more involved HTTP
client specific code.

> >
> > If you already have an HTTP 1.0 / 1.1 client implementation that is
> > licensed under the BSL (I see that libpion is already under the BSL)
> > and can be re-worked to use the basic_message implementation that's
> > already in cpp-netlib, then I think that would be a good start.
>
> I agree that this sounds like a good place to start.  In particular,
> with the latest release we really cleaned-up and broke out the HTTP
> parsing code (which I should say was largely based on the code in
> Christopher's ASIO http_server examples) so that it can be used for
> either requests or responses, and can even be used without networking
> code (our intention is to use it also with an HTTP sniffer).
>
> Your basic_message design is certainly cleaner and more extensible
> than our own.  We basically just have HTTPRequest and HTTPResponse
> classes that extend a base HTTPMessage class (with headers, version,
> etc).  We should be able to rework the parser so that it uses your
> classes instead, and be able to share that code between the two
> projects.
>

Sweet! Sounds like a good plan to me. We can keep the discussion here.

I plan on moving the code we currently have in the cpp-netlib project
into the sandbox, so that we can have more people who are interested
be able to work with the code we currently have. Although it's not
much, the foundation of most of the network-related stuff that I plan
on building on in the future is already available -- the
basic_message<> template has a very simple and extensible interface
which allows building algorithms around it simple and extensible.

> >
> > Yes, it's going to be a lot of work, and I still need all the help I
> > can get -- with the day job taking most of the time and attention from
> > me, it's going to be hard to put in time to code all the stuff that's
> > still in my head wanting to get out someday. I plan on putting in a
> > lot more time during the holidays here in the Philippines (that's
> > between the 25th and the 1st of January) to implement more of the HTTP
> > operations and the unit tests that need to cover the implementations
> > that need to be written. So yes, it's years of man-hours of work, and
> > there's plenty for everyone who's interested. :D
>
> Unfortunately, I'm really pressed for time as well and will probably
> not be able to do much over the next couple months.  Might be able to
> find some time though to get the ball rolling.
>

We can ask for help from people who are also interested in actually
getting this into fruition (and have the time and spare brain cycles
to be able to work on it in a volunteer effort).

Does moving the development to the boost sandbox make more sense? I
don't mind doing this, and since I'm not also personally able to
maximize the resources available from sourceforge, maybe having it in
the sandbox open it up to more people and more contributions?

Insights would be most appreciated.

--
Dean Michael C. Berris
Software Engineer, Friendster, Inc.
[http://cplusplus-soup.blogspot.com/]
[[hidden email]]
[+63 928 7291459]
[+1 408 4049523]
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Dean Michael Berris
In reply to this post by Michael Dickey
Hi Mike,

On Dec 11, 2007 4:28 AM, Michael Dickey <[hidden email]> wrote:
> Dean,
>
> I think we could combine cpp-netlib's messaging classes with pion's
> HTTP parsing classes to establish a basis for HTTP processing and
> provide client-side support for HTTP/1.0 and 1.1.
>

Great!

> There's two possible approaches to this that I see, one is to merge a
> "server" space into the Network library:
>
> boost::net::basic_message<>
> boost::net::http::request : public basic_message<>
> boost::net::http::response : public basic_message<>
> boost::net::server::tcp
> boost::net::server::http : public tcp
>
> Another would be to create an independent "server" library that uses
> the Network library classes:
>
> boost::server::tcp
> boost::server::http : public tcp
>
> They both seem like sensible approaches to me.  In either case, I
> think that HTTP would be a good starting focus for both, to get a
> working foundation incorporated into boost.  We could always submit
> additional protocols independently for review as their implementations
> are finished.
>
> What do you think?
>

I like the first one, where server:: is inside boost::net:: -- it at
least makes more sense that way, and if boost::server would be more
convenient, it can be pulled in appropriately with 'using
net::server'.

Would you mind committing the changes into the cpp-netlib subversion
project first, then perhaps later we go ahead and put it into the
boost sandbox?

--
Dean Michael C. Berris
Software Engineer, Friendster, Inc.
[http://cplusplus-soup.blogspot.com/]
[[hidden email]]
[+63 928 7291459]
[+1 408 4049523]
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Boost.HTTP ?

Dean Michael Berris
In reply to this post by Andrey Tcherepanov
Hi Andrey,

On Dec 11, 2007 5:00 AM, Andrey Tcherepanov <[hidden email]> wrote:
>
> I like the idea of basic_message<> (which could be used for SMTP and gods
> know what else as well), but what about UDP servers or, generally
> speaking, multi-transport protocols?
>

basic_message<> is intended to be divorced from the actual transport
being used to put the contents of it out the wire. The idea of
basic_message is primarily as a common structure -- or the foundation
-- on which all operations lower level (network transport,
serialization/deserialization, etc.) and higher level (protocol
specific transformations, chaining, stream operations, etc.) will deal
with. If I only had the time and talent, I'll come up with a visual
representation as to how the basic_message<> fits in all this
network-related code with a simple layered design. For now, this is
how I envision it:

Adapters/Transformations Layer
----------------------------------------------------
| basic_message<> and derivatives |
----------------------------------------------------
Transport Layer

For a client protocol 'stack', there will be protocol specific
adapters and transformations higher than the basic_message<>
abstraction and protocol specific derivatives. So the idea would then
look like (for HTTP):

HTTP Transformations/Adaptations Layer
------------------------------------------------------------
| http_request / http_response
| both deriving from basic_message<>
------------------------------------------------------------
Generic/HTTP Transport Layer

Question might be, what the transformations/adaptations layer would
contain. There is no straight answer, but what comes to mind are the
following:

- Stream Adapters (create an HTTP Session "Stream", cookie support, ETags, etc.)
- Iterator Adapters (turn a message or set of message(s) into
iterator-accessible objects for STL friendliness; further examples can
be imagined if there was more time to do it)
- Protocol-Protocol transformations (example, HTTP<-->SMTP, or
HTTP<-->XMPP, etc.)

And the possibilities are endless. ;-)

> Although http is sitting on top of TCP, there is an RTSP protocol which
> claims to be derived from HTTP/1.1 in most parts (well, at least RFC has
> lots of backrefs to HTTP spec), but allows to use connectionless transport
> (aka UDP). Moreover, RTSP treats Clients as something that can receive
> asynchronous stream change notifications (if allowed). I am not arguing
> that your design will not cover 95% of useful cases, but I think protocol
> parsing and transport layer might need more separation. This way if brave
> person will decide to implement RTSP (for example) as derived from HTTP,
> he/she will not be limited by TCP transport.
>

It will be divorced -- the concept of messages have nothing to do with
the actual transport and processing that happens in a specific
protocol stack.

I hope this helps! :-)

--
Dean Michael C. Berris
Software Engineer, Friendster, Inc.
[http://cplusplus-soup.blogspot.com/]
[[hidden email]]
[+63 928 7291459]
[+1 408 4049523]
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
12
Loading...