New JSON library from the maker of Beast!

classic Classic list List threaded Threaded
78 messages Options
1234
Reply | Threaded
Open this post in threaded view
|

New JSON library from the maker of Beast!

Boost - Dev mailing list
I've been working on a massively-multiplayer online blackjack casino
server, called Beast Lounge [1]. The server and client communicate
using JSON-RPC over WebSocket. I have developed a brand-new JSON
library for this project, in accordance with the following design
goals:

* Robust support for custom allocators throughout.
* Array and object interfaces closely track their corresponding C++20
container equivalents.
* Use `std::basic_string` for strings.
* Minimize use of templates for reduced compilation times.
* Parsers and serializers work incrementally (['online algorithms]).
* Elements in objects may also be iterated in insertion order.

You can see the JSON library in development here:

https://github.com/vinniefalco/json

Is there any interest in proposing this for Boost?

I'm happy to hear feedback or answer questions about this library.
Feel free to open an issue on the repository, or reply here.

Thanks

[1] <https://github.com/vinniefalco/BeastLounge>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
Hi Boost,

it looks great indeed.

A few questions:
- can you provide a point-by-point comparison with this other great
library: https://github.com/nlohmann/json?
- would it be able to interact with property_tree too?

Thanks
David

On Mon, Sep 23, 2019 at 4:06 AM Vinnie Falco via Boost <
[hidden email]> wrote:

> I've been working on a massively-multiplayer online blackjack casino
> server, called Beast Lounge [1]. The server and client communicate
> using JSON-RPC over WebSocket. I have developed a brand-new JSON
> library for this project, in accordance with the following design
> goals:
>
> * Robust support for custom allocators throughout.
> * Array and object interfaces closely track their corresponding C++20
> container equivalents.
> * Use `std::basic_string` for strings.
> * Minimize use of templates for reduced compilation times.
> * Parsers and serializers work incrementally (['online algorithms]).
> * Elements in objects may also be iterated in insertion order.
>
> You can see the JSON library in development here:
>
> https://github.com/vinniefalco/json
>
> Is there any interest in proposing this for Boost?
>
> I'm happy to hear feedback or answer questions about this library.
> Feel free to open an issue on the repository, or reply here.
>
> Thanks
>
> [1] <https://github.com/vinniefalco/BeastLounge>
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Sun, Sep 22, 2019 at 12:06 PM Vinnie Falco via Boost <
[hidden email]> wrote:

> I've been working on a massively-multiplayer online blackjack casino
> server, called Beast Lounge [1]. The server and client communicate
> using JSON-RPC over WebSocket. I have developed a brand-new JSON
> library for this project, in accordance with the following design
> goals:
>
> * Robust support for custom allocators throughout.
> * Array and object interfaces closely track their corresponding C++20
> container equivalents.
> * Use `std::basic_string` for strings.
> * Minimize use of templates for reduced compilation times.
> * Parsers and serializers work incrementally (['online algorithms]).
> * Elements in objects may also be iterated in insertion order.
>
> You can see the JSON library in development here:
>
> https://github.com/vinniefalco/json
>
> Is there any interest in proposing this for Boost?
>

I'm very interested; except...

I'm happy to hear feedback or answer questions about this library.
> Feel free to open an issue on the repository, or reply here.
>

Why does it need to depend on Beast? That's not the kind of dependency I'd
expect from a JSON library.

--
-- Rene Rivera
-- Grafik - Don't Assume Anything
-- Robot Dreams - http://robot-dreams.net

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
On Sun, Sep 22, 2019 at 6:12 PM Rene Rivera <[hidden email]> wrote:
> Why does it need to depend on Beast?
> That's not the kind of dependency I'd expect from a JSON library.

There are these four Beast includes which are leftovers from the
initial development which will eventually be removed so that the
library only depends on basic Boost facilities (Align, Assert, Config,
Container, Utility):

#include <boost/beast/core/static_string.hpp>
#include <boost/beast/core/file.hpp>
#include <boost/beast/core/flat_buffer.hpp>
#include <boost/beast/core/detail/clamp.hpp>

static_string is being developed into its own proposed library:

<https://github.com/vinniefalco/fixed_string>

clamp.hpp provides a trivial inline function that I will simply copy
over. The flat_buffer will be replaced with a straightforward memory
allocation. The dependency on file.hpp will be removed by using std
facilities to read from the file.

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Sun, Sep 22, 2019 at 5:46 PM David Bellot via Boost
<[hidden email]> wrote:
> - can you provide a point-by-point comparison with this other great
> library: https://github.com/nlohmann/json?

Features in nlohmann but not planned for Boost.JSON:

* JSON Pointer
* JSON Patch
* Specializing enum conversion
* Binary formats (BSON, CBOR, MessagePack, and UBJSON)
* Output format control
* A bunch of syntactic sugar

Features in Boost.JSON but not in nlohmann:

* Full allocator support
  To my knowledge, nlohmann JSON only supports allocators which are
DefaultConstructible. Boost.JSON gives full control over allocation
and uses a type-erased, reference counted allocator almost identical
to the polymorphic allocators in the standard. The library enforces a
simple invariant: All elements which are part of the same JSON
"document" (i.e. all share a common ancestor) are guaranteed to use
the same. This is enforced at runtime: when an element which uses a
different allocator is moved into a container, it is instead copied to
use the same allocator as the container (and thus the enclosing JSON
document).

* Object elements are also sorted on insertion order:
Iterating a JSON value of object type visits the elements in the order
of insertion.
<https://github.com/vinniefalco/json/blob/25ddea5f4d088f3e56911bf9f97549eeb938b9b5/include/boost/json/object.hpp#L30>

* Insert order control, e.g
<https://github.com/vinniefalco/json/blob/25ddea5f4d088f3e56911bf9f97549eeb938b9b5/include/boost/json/object.hpp#L247>

* C++20 conforming container APIs
  Including node_type, remove values from an object and reinsert them
elsewhere, or live outside the container:
<https://github.com/vinniefalco/json/blob/25ddea5f4d088f3e56911bf9f97549eeb938b9b5/include/boost/json/object.hpp#L312>

* Elements are implemented using non-template classes, most of the
function definitions may be compiled and linked through a static or
shared library. A header-only option is also available (via
configuration macro)

* The parser and serializer are designed to work incrementally.
However, the parser interface allows the caller to inform the parser
when the entirety (or remainder) of the JSON document is present. This
enables the parser to use a different algorithm when it is known ahead
of time that the entire document is available (currently, the
implementation does not take advantage of this feature but it is
possible in the future).

* The parser uses error codes. The implementation does not allow
exceptions to be raised from untrusted inputs.

* User-defined types can be exchanged to and from JSON value objects in 3 ways:
  - by writing a free function which is found via ADL for the type,
  - by adding member functions with conforming signatures for class types, or
  - by specializing a class template for the type
Here is an example of specialization to allow Boost.Asio IP address to
be constructed from a JSON value:
<https://github.com/vinniefalco/BeastLounge/blob/e4b085d3523047ffe8fa0c910592234da2aaea34/server/server.cpp#L53>

There is still some implementation left in my library so it should be
considered as an early beta software. BeastLounge runs on it, you can
see example code that uses it here:

<https://github.com/vinniefalco/BeastLounge/blob/e4b085d3523047ffe8fa0c910592234da2aaea34/server/rpc.cpp#L128>

With respect to nlohmann, the initial version 1.0 of Boost.JSON will
certainly not be as feature rich. Instead, I plan to focus on the
areas where I think it can offer things lacking in other JSON
libraries and do so with the high standards the community has come to
expect from Boost libraries.

> - would it be able to interact with property_tree too?

I don't plan on introducing a dependency on Boost.PropertyTree. The
parser is structured the same way as the Beast HTTP parser. The
algorithm is encoded into an abstract base class which is subclassed
to output into the JSON container. If desired, you can subclass the
parser yourself to store the results in a property tree:
<https://github.com/vinniefalco/json/blob/25ddea5f4d088f3e56911bf9f97549eeb938b9b5/include/boost/json/basic_parser.hpp#L157>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
On Sun, Sep 22, 2019 at 6:44 PM Vinnie Falco <[hidden email]> wrote:
> ...

Errata, the sentence should read:

"The library enforces a simple invariant: All elements which are part
of the same JSON
"document" (i.e. all share a common ancestor) are guaranteed to use
the same allocator."

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Sun, Sep 22, 2019 at 8:06 PM Vinnie Falco via Boost <
[hidden email]> wrote:

> [...] I have developed a brand-new JSON library for this project,

in accordance with the following design goals:
>
> * Robust support for custom allocators throughout.
> * Array and object interfaces closely track their corresponding C++20
> container equivalents.
> * Use `std::basic_string` for strings.
> * Minimize use of templates for reduced compilation times.
> * Parsers and serializers work incrementally (['online algorithms]).
> * Elements in objects may also be iterated in insertion order.
>

Hi,

What about performance?
Have you heard of https://github.com/lemire/simdjson?
Where would your library fall in that benchmark at the above link?

The already mentioned and popular nlohmann/json has a
convenient API, but fairs poorly in that benchmark for example.

Also, in client/server communications, it's less often a few huge JSON
documents, but rather lots of small documents, so the constant "startup"
time of the parser matters too, and in that same vein, a PULL-parser that
allows to build the native data-structures directly, rather than the
DOM-like
approach of fully converting the document to a built-in JSON object, and
then convert that to the native-structure, avoids the temp document, which
is especially useful for large documents.

Finally, there are many corner cases in JSON parsing. Is there the
equivalent
of Autobabhn for WebSocket, but for JSON parsing? Any plans to integrate
with such infrastructure, assuming there's one?

Thanks --DD

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
On Mon, Sep 23, 2019 at 5:19 AM Dominique Devienne via Boost
<[hidden email]> wrote:
> Hi,
>
> What about performance?
> Have you heard of https://github.com/lemire/simdjson?
> Where would your library fall in that benchmark at the above link?
> Finally, there are many corner cases in JSON parsing. Is there the
> equivalent
> of Autobabhn for WebSocket, but for JSON parsing? Any plans to integrate
> with such infrastructure, assuming there's one?

There's also https://github.com/miloyip/nativejson-benchmark which
benchmarks both performance and conformance.

> The already mentioned and popular nlohmann/json has a
> convenient API, but fairs poorly in that benchmark for example.
>
> Also, in client/server communications, it's less often a few huge JSON
> documents, but rather lots of small documents, so the constant "startup"
> time of the parser matters too, and in that same vein, a PULL-parser that
> allows to build the native data-structures directly, rather than the
> DOM-like
> approach of fully converting the document to a built-in JSON object, and
> then convert that to the native-structure, avoids the temp document, which
> is especially useful for large documents.
>

It looks like nlohmann/json now has this kind of API too:
1. https://tinyurl.com/nl-json-parse
2. https://tinyurl.com/nl-json-parse-callback

Glen

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
On Mon, Sep 23, 2019 at 6:44 AM Glen Fernandes  wrote:
> On Mon, Sep 23, 2019 at 5:19 AM Dominique Devienne via Boost

> > Also, in client/server communications, it's less often a few huge JSON
> > documents, but rather lots of small documents, so the constant "startup"
> > time of the parser matters too, and in that same vein, a PULL-parser that
> > allows to build the native data-structures directly, rather than the
> > DOM-like
> > approach of fully converting the document to a built-in JSON object, and
> > then convert that to the native-structure, avoids the temp document, which
> > is especially useful for large documents.
> >
>
> It looks like nlohmann/json now has this kind of API too:
> 1. https://tinyurl.com/nl-json-parse
> 2. https://tinyurl.com/nl-json-parse-callback
>

That is the use case that I find more interesting. If you want to
decouple your data representation in your  program from the
serialization format.

e.g. In memory if your data structure is something like:
    vector<map<string, pair<int, double> > >.

This could be expressed in json by something like:
    [   { "key": [1, 0.01],   "def": [9, 0.37], ... },
        { "xyz": [5, 1.25],   "abc": [2, 4.68], ... },
     ... ]

Your load it from some JSON file.  Your don't want to store/use it in
your program as a SomeLibrary::JsonArray. Past the point of
serialization it should be your own data structures.  Of course your
could always convert it from SomeLIbrary::JsonArray to your own
structures, but that's overhead you don't need if there's hundreds of
megabytes worth of content in that data.

Glen

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list


> On 23. Sep 2019, at 13:06, Glen Fernandes via Boost <[hidden email]> wrote:
>
> On Mon, Sep 23, 2019 at 6:44 AM Glen Fernandes  wrote:
>> On Mon, Sep 23, 2019 at 5:19 AM Dominique Devienne via Boost
>
>>> Also, in client/server communications, it's less often a few huge JSON
>>> documents, but rather lots of small documents, so the constant "startup"
>>> time of the parser matters too, and in that same vein, a PULL-parser that
>>> allows to build the native data-structures directly, rather than the
>>> DOM-like
>>> approach of fully converting the document to a built-in JSON object, and
>>> then convert that to the native-structure, avoids the temp document, which
>>> is especially useful for large documents.
>>>
>>
>> It looks like nlohmann/json now has this kind of API too:
>> 1. https://tinyurl.com/nl-json-parse
>> 2. https://tinyurl.com/nl-json-parse-callback
>>
>
> That is the use case that I find more interesting. If you want to
> decouple your data representation in your  program from the
> serialization format.
>
> e.g. In memory if your data structure is something like:
>    vector<map<string, pair<int, double> > >.
>
> This could be expressed in json by something like:
>    [   { "key": [1, 0.01],   "def": [9, 0.37], ... },
>        { "xyz": [5, 1.25],   "abc": [2, 4.68], ... },
>     ... ]
>
> Your load it from some JSON file.  Your don't want to store/use it in
> your program as a SomeLibrary::JsonArray. Past the point of
> serialization it should be your own data structures.  Of course your
> could always convert it from SomeLIbrary::JsonArray to your own
> structures, but that's overhead you don't need if there's hundreds of
> megabytes worth of content in that data.
>

Just throwing our library in the ring: https://github.com/taocpp/json

We have a SAX-style interface at the core, parsers only generate events, from there you can generate a DOM object or some direct UDT structure like mentioned above. (Uses some macros to register types). Or you directly pretty print it or convert to CBOR, MsgPack, etc.

You could even generate an nlohmann-DOM-value with our parsers. Benchmarks in the nativejson-benchmark showed that we were faster than nlohmann's builtin parser (that was some time ago, might have changed in the meantime)

Currently, our parser(s) do not support incremental parsing, though.

Daniel


> Glen
>
> _______________________________________________
> Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
On Mon, Sep 23, 2019 at 7:15 AM Daniel Frey wrote:
> Just throwing our library in the ring: https://github.com/taocpp/json
>
> We have a SAX-style interface at the core, parsers only generate events, from there you can generate a DOM object or some direct UDT structure like mentioned above. (Uses some macros to register types). Or you directly pretty print it or convert to CBOR, MsgPack, etc.
>
> You could even generate an nlohmann-DOM-value with our parsers. Benchmarks in the nativejson-benchmark showed that we were faster than nlohmann's builtin parser (that was some time ago, might have changed in the meantime)
>

Looks interesting.    Missing a Boost license though. :)

Glen

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list


> On 23. Sep 2019, at 13:27, Glen Fernandes <[hidden email]> wrote:
>
> On Mon, Sep 23, 2019 at 7:15 AM Daniel Frey wrote:
>> Just throwing our library in the ring: https://github.com/taocpp/json
>>
>> We have a SAX-style interface at the core, parsers only generate events, from there you can generate a DOM object or some direct UDT structure like mentioned above. (Uses some macros to register types). Or you directly pretty print it or convert to CBOR, MsgPack, etc.
>>
>> You could even generate an nlohmann-DOM-value with our parsers. Benchmarks in the nativejson-benchmark showed that we were faster than nlohmann's builtin parser (that was some time ago, might have changed in the meantime)
>>
>
> Looks interesting.    Missing a Boost license though. :)

Thanks, and making it dual-licensed is certainly not a problem if that's an actual problem. 😊

>
> Glen


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Mon, Sep 23, 2019 at 2:19 AM Dominique Devienne via Boost
<[hidden email]> wrote:
> What about performance?

I have not run any benchmarks, or invested any time in optimizing the
code. My main efforts thus far have been making sure that the public
interfaces are correct, and that things work.

> Have you heard of https://github.com/lemire/simdjson?

Yes I have seen that library. It is more of a proof-of-concept, and it
has the requirement that it requires the entire JSON documented to be
presented in a single contiguous input. For many use-cases, this
condition is acceptable. So I have designed the parser to be informed
when the condition is met, and use a different algorithm such as the
SIMD one above.

> a PULL-parser that allows to build the native data-structures directly,
> rather than the DOM-like approach of fully converting the document to
> a built-in JSON object

Yes, the JSON parser and the HTTP parser (in Beast) both are
implemented with an event-style interface and do not make assumptions
about the container used to store the data (this is provided by a
subclass). See:

<https://github.com/vinniefalco/json/blob/25ddea5f4d088f3e56911bf9f97549eeb938b9b5/include/boost/json/basic_parser.hpp#L28>
<https://github.com/vinniefalco/json/blob/25ddea5f4d088f3e56911bf9f97549eeb938b9b5/include/boost/json/parser.hpp#L24>

> Finally, there are many corner cases in JSON parsing. Is there the
> equivalent of Autobabhn for WebSocket, but for JSON parsing?

No idea, but if anyone knows of such a facility please let me know :)

> Any plans to integrate with such infrastructure, assuming there's one?

Sure!

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Mon, Sep 23, 2019 at 4:06 AM Glen Fernandes via Boost
<[hidden email]> wrote:
> Your don't want to store/use it in
> your program as a SomeLibrary::JsonArray. Past the point of
> serialization it should be your own data structures.

This is accomplished by making your own class derived from
`json::basic_parser`, and implementing the abstract virtual "event"
members:

<https://github.com/vinniefalco/json/blob/25ddea5f4d088f3e56911bf9f97549eeb938b9b5/include/boost/json/basic_parser.hpp#L30>

Thanks

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
Hi Vinnie!
I know that it is not standard JSON, but do you plan to support comments
(// ..., /* ... */)? It would be useful for storing configuration with
comments, for example.

Thanks,
Jarda

po 23. 9. 2019 v 15:36 odesílatel Vinnie Falco via Boost <
[hidden email]> napsal:

> On Mon, Sep 23, 2019 at 4:06 AM Glen Fernandes via Boost
> <[hidden email]> wrote:
> > Your don't want to store/use it in
> > your program as a SomeLibrary::JsonArray. Past the point of
> > serialization it should be your own data structures.
>
> This is accomplished by making your own class derived from
> `json::basic_parser`, and implementing the abstract virtual "event"
> members:
>
> <
> https://github.com/vinniefalco/json/blob/25ddea5f4d088f3e56911bf9f97549eeb938b9b5/include/boost/json/basic_parser.hpp#L30
> >
>
> Thanks
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
Hi Vinnie,

Vinnie Falco wrote:
> I've been working on a massively-multiplayer online blackjack casino
> server, called Beast Lounge [1]. The server and client communicate
> using JSON-RPC over WebSocket. I have developed a brand-new JSON
> library for this project, in accordance with the following design
> goals:

I am reminded of the various discussions of alternative styles of
XML parsers that have happened on this list over the years.  People
have a surprising variety of often-conflicting requirements or
preferences.  I think it's unlikely that any one solution will suit
everyone - but maybe there are common bits of functionality that
can be shared?

My preference has always been for parsing by memory-mapping the entire
file, or equivalently reading the entire document into memory as a blob
of text, and then providing iterators that advance through the text
looking for the next element, attribute, character etc.  I think one of
the first XML parsers to work this way was RapidXML.  Their aim was to
get the parsing speed as close to strlen() as possible and they did
pretty well.  Others have mentioned some benchmarks and I encourage
you to try them - and/or at least be clear about whether performance is
a design goal.


Regards, Phil.





_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Mon, Sep 23, 2019 at 7:51 AM JF via Boost <[hidden email]> wrote:
> I know that it is not standard JSON, but do you plan to support comments
> (// ..., /* ... */)? It would be useful for storing configuration

I used the same approach with the JSON parser that I used with Beast's
HTTP parser. That is, a strict parser which strives to adhere to the
letter of the spec. This does favor the use-case for networking
purposes (since parser inputs are often from untrusted sources) and
disadvantage the configuration file case. However, there are already
lots of nice libraries (including JSON) for processing configuration
files, but no JSON libraries that are specifically tuned for
networking (allocator and incremental parsing features). So comments
are not supported, and I wasn't thinking about adding them. Although
alternate parser implementations could be something I explore in the
future.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On Mon, Sep 23, 2019 at 8:17 AM Phil Endecott via Boost
<[hidden email]> wrote:
> but maybe there are common bits of functionality that
> can be shared?

Perhaps, but my preference is for monolithic parsers with no external
dependencies and little to no configurability. They are easier to
audit (and cheaper), and I do plan to commission a security review for
Boost.JSON as I have done with Beast:

<https://vinniefalco.github.io/BeastAssets/Beast%20-%20Hybrid%20Application%20Assessment%202017%20-%20Assessment%20Report%20-%2020171114.pdf>

It is also easier to maintain, and less likely to require changes
(which bring the risk of new vulnerabilities) if the scope of
functionality is strictly limited. It is true that this results in a
parser which is less flexible. A survey of parsers in existing JSON
libraries shows great diversity, so there is no shortage of
flexibility there. I think there is room for one more strict parser
with a static set of features.

> My preference has always been for parsing by memory-mapping the entire
> file, or equivalently reading the entire document into memory as a blob
> of text

While parsing is important, as with the HTTP it is the least
interesting aspect of JSON since parsing happens only once but
inspection and modification of a JSON document (the
`boost::json::value` type) happens continually, including across
library API boundaries where JSON value types appear in function
signatures or data members.

> ...be clear about whether performance is  a design goal.

Performance is a design goal, but it is with respect to performance in
the larger context of a network application. This library is less
concerned about parsing a large chunk of in-memory serialized JSON
over and over again inside a tight loop to hit a meaningless number in
a contrived benchmark, and more concerned about ensuring that network
programs have control over how and when memory allocation takes place,
latency, and resource fairness when handling a large number of
connections. That is why the library is built from the ground up to
support allocators, to support incremental operation for parsing and
serialization (bounded work in each I/O cycle reduces latency and
increases fairness),

Since the parser is presented with one or more buffers of memory
containing the JSON document, and there is an API to inform the parser
when these memory buffers represent the complete document, it should
be possible to apply most of the optimizations currently used in other
libraries, including SIMD algorithms when the complete document is
presented. That said, if the experience with HTTP in Beast is
representative of network applications which use JSON (a reasonable
assumption), relatively little time is spent parsing a JSON RPC
command coming from a connection compared to the time required to
process the command, so the gains to be had from an optimized parser
may not be so impressive. I will still eventually apply optimizations
to it of course, for bragging rights. But I am in no hurry.

Regards

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 9/23/19 5:16 PM, Phil Endecott via Boost wrote:

> I am reminded of the various discussions of alternative styles of
> XML parsers that have happened on this list over the years.  People
> have a surprising variety of often-conflicting requirements or
> preferences.  I think it's unlikely that any one solution will suit
> everyone - but maybe there are common bits of functionality that
> can be shared?

As a former developer of one of said XML parsers, we learned the
proper abstractions the hard way. If you start with a pull parser (what
Vinnie refers to as an online parser, and what you refer to as
an iterating parser), such as the XmlTextReader, then all the other
interfaces flows naturally from that.

Although the pull parser is mainly used as the basic building block
for the other abstractions, it can also be used directly, e.g. for quick
scanning of large JSON documents without memory allocation.

A push parser (SAX) can easily be created by calling the pull parser
in a loop and firing off events.

Serialization is done by incrementally using a pull parser inside a
serialization input archive, and likewise a a similar interface for
generating the layout (e.g. XmlTextWriter) can be used for output
archives.

A tree parser (DOM) is simply a push parser that generates nodes as
events are fired off.

That is the design principles behind this JSON parser:

   http://breese.github.io/trial/protocol/

> My preference has always been for parsing by memory-mapping the entire
> file, or equivalently reading the entire document into memory as a blob
> of text, and then providing iterators that advance through the text
> looking for the next element, attribute, character etc.  I think one of
> the first XML parsers to work this way was RapidXML.  Their aim was to

The Microsoft XML parser came first.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: New JSON library from the maker of Beast!

Boost - Dev mailing list
In reply to this post by Boost - Dev mailing list
On 9/23/19 1:05 PM, Glen Fernandes via Boost wrote:

> That is the use case that I find more interesting. If you want to
> decouple your data representation in your  program from the
> serialization format.

Which is what Boost.Serialization archives gives us.

_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
1234