Any Interest in a port of ZLib to C++?

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
Beast currently has its own port of ZLib to C++:

<https://github.com/boostorg/beast/tree/develop/include/boost/beast/zlib>

Is there any interest in having this proposed as its own separate
Boost library (with a separate review, which has nothing to do with
Beast)? The differences would be as follows:

* Ported to use C++11 idioms
* Same header-only / compiled source model as Boost.JSON
* Better documentation and clear API
* Improved tests

Programs which use this C++ port of ZLib would get their zlib
dependency from Boost instead of the official zlib distribution. And
they can integrate it into their projects the same way that Boost
libraries are integrated. Or they can include <boost/zlib/src.hpp> in
any one translation unit (like JSON).

The motivation for this is twofold:

* Provide an easier way for users to access ZLib's functionality
* To use this in some new libraries that I'm going to propose

--
Regards,
Vinnie

Follow me on GitHub: https://github.com/vinniefalco

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
Le 2021-03-23 06:14, Vinnie Falco via Boost a écrit :
> Beast currently has its own port of ZLib to C++:
>
> <https://github.com/boostorg/beast/tree/develop/include/boost/beast/zlib>

I see it's a derivative work of zlib, rather than a from scratch
implementation. IANAL, but isn't there a license issue there? (zlib
license has non-mandatory aknowledgements requests which i'm not sure
are boost-compatible).

Regards,

Julien

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
I am not a lawyer either, but I believe that this would not clash with
the BSL. If the Zlib license *required* that you put an acknowledgement
in the documentation etc, that would clash. But the way it is worded, I
think it's compatible.

To answer the original question, I think this is a nice idea and I would
probably use it.

Kind regards,

Philip Bennefall


On 3/23/2021 8:55 AM, Julien Blanc via Boost wrote:

> Le 2021-03-23 06:14, Vinnie Falco via Boost a écrit :
>> Beast currently has its own port of ZLib to C++:
>>
>> <https://github.com/boostorg/beast/tree/develop/include/boost/beast/zlib>
>>
>
> I see it's a derivative work of zlib, rather than a from scratch
> implementation. IANAL, but isn't there a license issue there? (zlib
> license has non-mandatory aknowledgements requests which i'm not sure
> are boost-compatible).
>
> Regards,
>
> Julien
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Hi Vinnie,

The library you propose is really nice to have !

I had tried to use zlib with boost asio stream.
https://stackoverflow.com/questions/44712852/how-to-decompress-the-data-in-boostasiostreambuf-using-boostiostreamsfilt

However, I gave up on it and then I used the raw zlib directly.
https://github.com/msgpack/msgpack-c/blob/cpp_master/example/boost/asio_send_recv_zlib.cpp

If I would have modern, stream friendly, and header-only zlib library,
I would be very glad.

---
Thanks
Takatoshi

On Tue, Mar 23, 2021 at 2:16 PM Vinnie Falco via Boost
<[hidden email]> wrote:

>
> Beast currently has its own port of ZLib to C++:
>
> <https://github.com/boostorg/beast/tree/develop/include/boost/beast/zlib>
>
> Is there any interest in having this proposed as its own separate
> Boost library (with a separate review, which has nothing to do with
> Beast)? The differences would be as follows:
>
> * Ported to use C++11 idioms
> * Same header-only / compiled source model as Boost.JSON
> * Better documentation and clear API
> * Improved tests
>
> Programs which use this C++ port of ZLib would get their zlib
> dependency from Boost instead of the official zlib distribution. And
> they can integrate it into their projects the same way that Boost
> libraries are integrated. Or they can include <boost/zlib/src.hpp> in
> any one translation unit (like JSON).
>
> The motivation for this is twofold:
>
> * Provide an easier way for users to access ZLib's functionality
> * To use this in some new libraries that I'm going to propose
>
> --
> Regards,
> Vinnie
>
> Follow me on GitHub: https://github.com/vinniefalco
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Tue, 23 Mar 2021 at 05:14, Vinnie Falco via Boost
<[hidden email]> wrote:
>
> Beast currently has its own port of ZLib to C++:
>
> <https://github.com/boostorg/beast/tree/develop/include/boost/beast/zlib>
>
> Is there any interest in having this proposed as its own separate
> Boost library (with a separate review, which has nothing to do with
> Beast)? The differences would be as follows:

While there's no harm in this, is zlib still really relevant in 2021?
bz2, lzma, lz4, zstd are all superior.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Hi Vinnie,

I think this would be a good addition and something I'd probably use.

Regards,

Pete

On Tue, 23 Mar 2021 at 05:16, Vinnie Falco via Boost <[hidden email]>
wrote:

> Beast currently has its own port of ZLib to C++:
>
> <https://github.com/boostorg/beast/tree/develop/include/boost/beast/zlib>
>
> Is there any interest in having this proposed as its own separate
> Boost library (with a separate review, which has nothing to do with
> Beast)? The differences would be as follows:
>
> * Ported to use C++11 idioms
> * Same header-only / compiled source model as Boost.JSON
> * Better documentation and clear API
> * Improved tests
>
> Programs which use this C++ port of ZLib would get their zlib
> dependency from Boost instead of the official zlib distribution. And
> they can integrate it into their projects the same way that Boost
> libraries are integrated. Or they can include <boost/zlib/src.hpp> in
> any one translation unit (like JSON).
>
> The motivation for this is twofold:
>
> * Provide an easier way for users to access ZLib's functionality
> * To use this in some new libraries that I'm going to propose
>
> --
> Regards,
> Vinnie
>
> Follow me on GitHub: https://github.com/vinniefalco
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 3/23/21 8:14 AM, Vinnie Falco via Boost wrote:

> Beast currently has its own port of ZLib to C++:
>
> <https://github.com/boostorg/beast/tree/develop/include/boost/beast/zlib>
>
> Is there any interest in having this proposed as its own separate
> Boost library (with a separate review, which has nothing to do with
> Beast)? The differences would be as follows:
>
> * Ported to use C++11 idioms
> * Same header-only / compiled source model as Boost.JSON
> * Better documentation and clear API
> * Improved tests
>
> Programs which use this C++ port of ZLib would get their zlib
> dependency from Boost instead of the official zlib distribution. And
> they can integrate it into their projects the same way that Boost
> libraries are integrated. Or they can include <boost/zlib/src.hpp> in
> any one translation unit (like JSON).
>
> The motivation for this is twofold:
>
> * Provide an easier way for users to access ZLib's functionality
> * To use this in some new libraries that I'm going to propose

I would most probably *not* use it and rely on zlib or zlib-ng because
(a) it would probably be used by other libraries that are not part of
your ecosystem, (b) because its maintenance would be a someone else's
burden (e.g. a distro maintainer's) and not mine, and (c), especially in
zlib-ng case, I suspect it would be better optimized than your
header-only version.

Because of this I'd like to ask you to at least provide an option to
build your present and future libraries against zlib.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Tue, Mar 23, 2021 at 3:49 AM Mathias Gaunard
<[hidden email]> wrote:
> While there's no harm in this, is zlib still really relevant in 2021?
> bz2, lzma, lz4, zstd are all superior.

Great question. Yes the deflate algorithm is still relevant because it
is used in websocket's permessage-deflate implementation, and it is
also a very common Content-Encoding for HTTP message payloads.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 3/23/2021 1:14 AM, Vinnie Falco via Boost wrote:

> Beast currently has its own port of ZLib to C++:
>
> <https://github.com/boostorg/beast/tree/develop/include/boost/beast/zlib>
>
> Is there any interest in having this proposed as its own separate
> Boost library (with a separate review, which has nothing to do with
> Beast)? The differences would be as follows:
>
> * Ported to use C++11 idioms
> * Same header-only / compiled source model as Boost.JSON
> * Better documentation and clear API
> * Improved tests
>
> Programs which use this C++ port of ZLib would get their zlib
> dependency from Boost instead of the official zlib distribution. And
> they can integrate it into their projects the same way that Boost
> libraries are integrated. Or they can include <boost/zlib/src.hpp> in
> any one translation unit (like JSON).
>
> The motivation for this is twofold:
>
> * Provide an easier way for users to access ZLib's functionality
> * To use this in some new libraries that I'm going to propose
>

The official zlib at https://github.com/madler/zlib has 109 issues, 156
PRs, one of which is mine, so yes a Boost version would be welcome to me.


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 23.03.21 06:14, Vinnie Falco via Boost wrote:
> Is there any interest in having this proposed as its own separate
> Boost library (with a separate review, which has nothing to do with
> Beast)?

Better to have this out in the open than lurking in the belly of the Beast.

For me, the big question for a zlib derivative is where it stands in
relation to the original zlib in terms of bugs and security.


--
Rainer Deyke ([hidden email])


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
On Tue, Mar 23, 2021 at 1:34 PM Rainer Deyke via Boost
<[hidden email]> wrote:
> For me, the big question for a zlib derivative is where it stands in
> relation to the original zlib in terms of bugs and security.

Great question. So, the current implementation in Beast (which will
become the starting point for an independent library) started out as a
copy of the ZLib sources, followed by my changes to port it to
header-only C++. I changed very little although I did rework some
lines of code to be more idiomatic and concise using C++ features. I
also removed the gzip file format support since Beast websockets did
not need it, and it wouldn't have enough testing.

The plans for this library in terms of bugs and security include:

* Put back the gzip support by porting the relevant original zlib code again
* Add Drone CI, GitHub Actions CI for C++11 and later compilers (same
as Beast and JSON)
* Add Code Coverage
* Add Github Actions fuzzing
* Manually translate and apply each of the newer commits in the
upstream ZLib repository
* Refactor statements to be code-coverage friendly
* Add concise tests using coverage as guidance (100% coverage target)
* Add tests for existing zlib input vectors
* Add tests vectors for known attacks
* Other tests as deemed useful

Additional plans include:

* Benchmark program plus CI integration to detect performance regression
* Refactor public API to be idiomatic for C++
* Rewrite Javadocs to be readable
* Create new exposition to show how to use the library
* Add examples for common tasks

As with my other libraries, this will be submitted to Bishop Fox for a
periodic security review and report. In addition, there will be a
dedicated maintainer so that issues can be addressed quickly and the
library can be refined. Although I don't imagine there is much room
left for innovation beyond cosmetic improvements since ZLib is very
mature.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Tue, Mar 23, 2021 at 1:34 PM Rainer Deyke via Boost
<[hidden email]> wrote:
> ...

One change I forgot to mention, is that this independent zlib library
will use the new "included source" model for the header-only mode. In
other words you will have to include <boost/zlib/src.hpp> in exactly
one translation unit. Alternatively you can link with the library as
usual. This provides the ease of integration of header-only without
the accompanying compilation performance penalties. Boost.JSON uses
this model and it has been working well.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On di, 23. mrt 18:43, Takatoshi Kondo via Boost wrote:
> Hi Vinnie,
>
> The library you propose is really nice to have !
>
> I had tried to use zlib with boost asio stream.
> https://stackoverflow.com/questions/44712852/how-to-decompress-the-data-in-boostasiostreambuf-using-boostiostreamsfilt
>

I second that motion. I'll submit another case, where I recently
finally managed to get websocket per-message deflation working for
misbehaved services:

 - https://stackoverflow.com/questions/66282247/c-inconsistent-behavior-with-boost-streambuf-deflating/66285851#66285851 

If the ZLib interface were (part of) a documented library, then
perhaps it would have taken roughly a year to solve this problem
first asked here many months earlier
https://stackoverflow.com/questions/62496815/boost-inflate-algorithm-decompress/62505788#comment117189671_62505788

Cheers,
Seth

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
wt., 23 mar 2021 o 06:16 Vinnie Falco via Boost <[hidden email]>
napisał(a):

> Beast currently has its own port of ZLib to C++:
>
> <https://github.com/boostorg/beast/tree/develop/include/boost/beast/zlib>
>
> Is there any interest in having this proposed as its own separate
> Boost library (with a separate review, which has nothing to do with
> Beast)? The differences would be as follows:
>
> * Ported to use C++11 idioms
> * Same header-only / compiled source model as Boost.JSON
> * Better documentation and clear API
> * Improved tests
>
> Programs which use this C++ port of ZLib would get their zlib
> dependency from Boost instead of the official zlib distribution. And
> they can integrate it into their projects the same way that Boost
> libraries are integrated. Or they can include <boost/zlib/src.hpp> in
> any one translation unit (like JSON).
>

In my work we currently use ZLib. If at the point of making a decision
which library to use there was an option to choose the Boost version, we
would definitely go for it. For me, a library under Boost is like a "Boost
certificate that guarantees certain things:
* The documentation will not be just a list of classes and functions, but
will also tell me the limits of the library, how I am supposed to use it,
and how not to use it; what problems it solves, and what problems are
beyond its scope; how it will interact with other libraries.
* I know that the design has included (and documented) things like
exception safety and thread safety. (There are good libraries in the wild
that never mention if it is safe to call them from two threads).

Of course, if a library for compression is extracted from Beast and added
as a distinct library in Boost, I would expect that it comes with an
interface for compression libraries that makes sense for other compression
algorithm implementations.

Regards,
&rzej;


> The motivation for this is twofold:
>
> * Provide an easier way for users to access ZLib's functionality
> * To use this in some new libraries that I'm going to propose
>
> --
> Regards,
> Vinnie
>
> Follow me on GitHub: https://github.com/vinniefalco
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: Any Interest in a port of ZLib to C++?

Boost - Dev mailing list
Well, it looks like there is some interest :)

Truth be told, I didn't expect so much interest. This is moving forward!

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost