[Spirit] parsing double precision issues

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[Spirit] parsing double precision issues

Boost - Users mailing list
Hello,

I am continuing to work through my Protobuf parsing with
Constant/Floating-Point tests. Double precision is rearing its ugly
head and I am curious what's the best way to handle potentially
determining a healthy epsilon given the expected parsed value. For the
most part, I think it's going to work, but for the precision issues.
Catch:

  REQUIRE( actual == expected )
with expansion:
  struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
'opt_sign': null }
  ==
  struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
'opt_sign': null }
with messages:
  Source: syntax = 'proto2';option L = 1.16624e+306;
  Delta was: 2.35309e+300

There was clearly a delta involved, most like a precision issue
encountered during the parse. Without going into a great deal of
depth, if possible:

namespace my { namespace protobuf { namespace ast {
using float_t = double;
enum num_sign_t {
    sign_none
    /// '-'
    , sign_minus
    /// '+'
    , sign_plus
};

// Which includes
struct floating_point_t : numeric_t<float_t> { /* ... */ };
} } }

// Without sign, dot required.
// >>> TODO: TBD: Precision issue? Or an actual double parsing issue? <<<
float_lit %= real_parser<ast::float_t, strict_ureal_policies<ast::float_t>>{};
floating_point_lit %= -num_lit_sign >> float_lit;

qi::rule<It, ast::num_sign_t()> num_lit_sign;

qi::rule<It, ast::float_t()> float_lit;
qi::rule<It, ast::floating_point_t()> floating_point_lit;

https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#floating-point-literals
https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#constant
https://www.boost.org/doc/libs/1_68_0/libs/spirit/doc/html/spirit/qi/reference/numeric/real.html

Cheers,

Michael Powell
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: [Spirit] parsing double precision issues

Boost - Users mailing list
On Wed, Nov 14, 2018 at 1:58 PM Michael Powell <[hidden email]> wrote:
>
> Hello,
>
> I am continuing to work through my Protobuf parsing with
> Constant/Floating-Point tests. Double precision is rearing its ugly
> head and I am curious what's the best way to handle potentially
> determining a healthy epsilon given the expected parsed value. For the
> most part, I think it's going to work, but for the precision issues.
> Catch:

After doing some background reading on floating point comparison ...
it's not so simple as a==b, but I kind of knew that already. Even the
simple absolute difference epsilon check is not so robust. I did a
combination of nearly, relatively, essentially, etc, checks, and I
think I have gotten past that immediate issue.

Now I actually have a parsing issue... According to the Protobuf v2
spec, this should parse, I think. However, it is failing to parse.

  CHECK( parse_tried == expected_tried )
with expansion:
  false == true
with message:
  Source: syntax = 'proto2';option a =
933252761176835091419019014852578134454945897134240121039056292642745934626544776679309792643426912681734775655456099849022472225507009671635655261635494179939591695082743337844489696175391517474159955183289891072214770215979051082304343590361421008538352849939802258131368544983961387362613078430270357504.000000;
  REQUIRE( tried )
with expansion:
  false
with message:
  Source: syntax = 'proto2';option a =
933252761176835091419019014852578134454945897134240121039056292642745934626544776679309792643426912681734775655456099849022472225507009671635655261635494179939591695082743337844489696175391517474159955183289891072214770215979051082304343590361421008538352849939802258131368544983961387362613078430270357504.000000;

>   REQUIRE( actual == expected )
> with expansion:
>   struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
> 'opt_sign': null }
>   ==
>   struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
> 'opt_sign': null }
> with messages:
>   Source: syntax = 'proto2';option L = 1.16624e+306;
>   Delta was: 2.35309e+300
>
> There was clearly a delta involved, most like a precision issue
> encountered during the parse. Without going into a great deal of
> depth, if possible:
>
> namespace my { namespace protobuf { namespace ast {
> using float_t = double;
> enum num_sign_t {
>     sign_none
>     /// '-'
>     , sign_minus
>     /// '+'
>     , sign_plus
> };
>
> // Which includes
> struct floating_point_t : numeric_t<float_t> { /* ... */ };
> } } }
>
> // Without sign, dot required.
> // >>> TODO: TBD: Precision issue? Or an actual double parsing issue? <<<
> float_lit %= real_parser<ast::float_t, strict_ureal_policies<ast::float_t>>{};
> floating_point_lit %= -num_lit_sign >> float_lit;
>
> qi::rule<It, ast::num_sign_t()> num_lit_sign;
>
> qi::rule<It, ast::float_t()> float_lit;
> qi::rule<It, ast::floating_point_t()> floating_point_lit;
>
> https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#floating-point-literals
> https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#constant
> https://www.boost.org/doc/libs/1_68_0/libs/spirit/doc/html/spirit/qi/reference/numeric/real.html
>
> Cheers,
>
> Michael Powell
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: [Spirit] parsing double precision issues

Boost - Users mailing list
On Wed, Nov 14, 2018 at 5:05 PM Michael Powell <[hidden email]> wrote:

>
> On Wed, Nov 14, 2018 at 1:58 PM Michael Powell <[hidden email]> wrote:
> >
> > Hello,
> >
> > I am continuing to work through my Protobuf parsing with
> > Constant/Floating-Point tests. Double precision is rearing its ugly
> > head and I am curious what's the best way to handle potentially
> > determining a healthy epsilon given the expected parsed value. For the
> > most part, I think it's going to work, but for the precision issues.
> > Catch:
>
> After doing some background reading on floating point comparison ...
> it's not so simple as a==b, but I kind of knew that already. Even the
> simple absolute difference epsilon check is not so robust. I did a
> combination of nearly, relatively, essentially, etc, checks, and I
> think I have gotten past that immediate issue.
>
> Now I actually have a parsing issue... According to the Protobuf v2
> spec, this should parse, I think. However, it is failing to parse.
>
>   CHECK( parse_tried == expected_tried )
> with expansion:
>   false == true
> with message:
>   Source: syntax = 'proto2';option a =
> 933252761176835091419019014852578134454945897134240121039056292642745934626544776679309792643426912681734775655456099849022472225507009671635655261635494179939591695082743337844489696175391517474159955183289891072214770215979051082304343590361421008538352849939802258131368544983961387362613078430270357504.000000;
>   REQUIRE( tried )
> with expansion:
>   false
> with message:
>   Source: syntax = 'proto2';option a =
> 933252761176835091419019014852578134454945897134240121039056292642745934626544776679309792643426912681734775655456099849022472225507009671635655261635494179939591695082743337844489696175391517474159955183289891072214770215979051082304343590361421008538352849939802258131368544983961387362613078430270357504.000000;

This much I can say at the moment, it does not appear to be a max
double issue, literally:

179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368.0000000000000000

That would stand to reason, though, since my test data is randomly
generated within max double boundaries.

I am scratching my head on the strict real policies, however. If the
docs are accurate, I do not see why that would not accept the fixed
double format...

But it could be a concern that parsing is done apart from Qi rules. It
seems like some rigid logic embedded there.

https://www.boost.org/doc/libs/1_68_0/libs/spirit/doc/html/spirit/qi/reference/numeric/real.html

> >   REQUIRE( actual == expected )
> > with expansion:
> >   struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
> > 'opt_sign': null }
> >   ==
> >   struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
> > 'opt_sign': null }
> > with messages:
> >   Source: syntax = 'proto2';option L = 1.16624e+306;
> >   Delta was: 2.35309e+300
> >
> > There was clearly a delta involved, most like a precision issue
> > encountered during the parse. Without going into a great deal of
> > depth, if possible:
> >
> > namespace my { namespace protobuf { namespace ast {
> > using float_t = double;
> > enum num_sign_t {
> >     sign_none
> >     /// '-'
> >     , sign_minus
> >     /// '+'
> >     , sign_plus
> > };
> >
> > // Which includes
> > struct floating_point_t : numeric_t<float_t> { /* ... */ };
> > } } }
> >
> > // Without sign, dot required.
> > // >>> TODO: TBD: Precision issue? Or an actual double parsing issue? <<<
> > float_lit %= real_parser<ast::float_t, strict_ureal_policies<ast::float_t>>{};
> > floating_point_lit %= -num_lit_sign >> float_lit;
> >
> > qi::rule<It, ast::num_sign_t()> num_lit_sign;
> >
> > qi::rule<It, ast::float_t()> float_lit;
> > qi::rule<It, ast::floating_point_t()> floating_point_lit;
> >
> > https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#floating-point-literals
> > https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#constant
> > https://www.boost.org/doc/libs/1_68_0/libs/spirit/doc/html/spirit/qi/reference/numeric/real.html
> >
> > Cheers,
> >
> > Michael Powell
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: [Spirit] parsing double precision issues

Boost - Users mailing list
On Wed, Nov 14, 2018 at 6:35 PM Michael Powell <[hidden email]> wrote:

>
> On Wed, Nov 14, 2018 at 5:05 PM Michael Powell <[hidden email]> wrote:
> >
> > On Wed, Nov 14, 2018 at 1:58 PM Michael Powell <[hidden email]> wrote:
> > >
> > > Hello,
> > >
> > > I am continuing to work through my Protobuf parsing with
> > > Constant/Floating-Point tests. Double precision is rearing its ugly
> > > head and I am curious what's the best way to handle potentially
> > > determining a healthy epsilon given the expected parsed value. For the
> > > most part, I think it's going to work, but for the precision issues.
> > > Catch:

No response? Does anyone have any insights as to the real parser policies?

The documented grammar would lead me to believe that "fixed format",
literally, (*digits  >> '.' >> +digits) | (+digits  >> '.' >>
*digits), in Qi phraseology, might be possible. However, I am finding
potentially this is not the case.

> > After doing some background reading on floating point comparison ...
> > it's not so simple as a==b, but I kind of knew that already. Even the
> > simple absolute difference epsilon check is not so robust. I did a
> > combination of nearly, relatively, essentially, etc, checks, and I
> > think I have gotten past that immediate issue.
> >
> > Now I actually have a parsing issue... According to the Protobuf v2
> > spec, this should parse, I think. However, it is failing to parse.
> >
> >   CHECK( parse_tried == expected_tried )
> > with expansion:
> >   false == true
> > with message:
> >   Source: syntax = 'proto2';option a =
> > 933252761176835091419019014852578134454945897134240121039056292642745934626544776679309792643426912681734775655456099849022472225507009671635655261635494179939591695082743337844489696175391517474159955183289891072214770215979051082304343590361421008538352849939802258131368544983961387362613078430270357504.000000;
> >   REQUIRE( tried )
> > with expansion:
> >   false
> > with message:
> >   Source: syntax = 'proto2';option a =
> > 933252761176835091419019014852578134454945897134240121039056292642745934626544776679309792643426912681734775655456099849022472225507009671635655261635494179939591695082743337844489696175391517474159955183289891072214770215979051082304343590361421008538352849939802258131368544983961387362613078430270357504.000000;
>
> This much I can say at the moment, it does not appear to be a max
> double issue, literally:
>
> 179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368.0000000000000000
>
> That would stand to reason, though, since my test data is randomly
> generated within max double boundaries.
>
> I am scratching my head on the strict real policies, however. If the
> docs are accurate, I do not see why that would not accept the fixed
> double format...
>
> But it could be a concern that parsing is done apart from Qi rules. It
> seems like some rigid logic embedded there.
>
> https://www.boost.org/doc/libs/1_68_0/libs/spirit/doc/html/spirit/qi/reference/numeric/real.html
>
> > >   REQUIRE( actual == expected )
> > > with expansion:
> > >   struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
> > > 'opt_sign': null }
> > >   ==
> > >   struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
> > > 'opt_sign': null }
> > > with messages:
> > >   Source: syntax = 'proto2';option L = 1.16624e+306;
> > >   Delta was: 2.35309e+300
> > >
> > > There was clearly a delta involved, most like a precision issue
> > > encountered during the parse. Without going into a great deal of
> > > depth, if possible:
> > >
> > > namespace my { namespace protobuf { namespace ast {
> > > using float_t = double;
> > > enum num_sign_t {
> > >     sign_none
> > >     /// '-'
> > >     , sign_minus
> > >     /// '+'
> > >     , sign_plus
> > > };
> > >
> > > // Which includes
> > > struct floating_point_t : numeric_t<float_t> { /* ... */ };
> > > } } }
> > >
> > > // Without sign, dot required.
> > > // >>> TODO: TBD: Precision issue? Or an actual double parsing issue? <<<
> > > float_lit %= real_parser<ast::float_t, strict_ureal_policies<ast::float_t>>{};
> > > floating_point_lit %= -num_lit_sign >> float_lit;
> > >
> > > qi::rule<It, ast::num_sign_t()> num_lit_sign;
> > >
> > > qi::rule<It, ast::float_t()> float_lit;
> > > qi::rule<It, ast::floating_point_t()> floating_point_lit;
> > >
> > > https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#floating-point-literals
> > > https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#constant
> > > https://www.boost.org/doc/libs/1_68_0/libs/spirit/doc/html/spirit/qi/reference/numeric/real.html
> > >
> > > Cheers,
> > >
> > > Michael Powell
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|

Re: [Spirit] parsing double precision issues

Boost - Users mailing list
Quite possibly this is related to an issue reported some three years
ago. How did it break, first of all? Second, why is it still broken.
Today what is clear is that it does not "just work" any longer.

https://svn.boost.org/trac10/ticket/11608
On Sat, Nov 17, 2018 at 3:38 PM Michael Powell <[hidden email]> wrote:

>
> On Wed, Nov 14, 2018 at 6:35 PM Michael Powell <[hidden email]> wrote:
> >
> > On Wed, Nov 14, 2018 at 5:05 PM Michael Powell <[hidden email]> wrote:
> > >
> > > On Wed, Nov 14, 2018 at 1:58 PM Michael Powell <[hidden email]> wrote:
> > > >
> > > > Hello,
> > > >
> > > > I am continuing to work through my Protobuf parsing with
> > > > Constant/Floating-Point tests. Double precision is rearing its ugly
> > > > head and I am curious what's the best way to handle potentially
> > > > determining a healthy epsilon given the expected parsed value. For the
> > > > most part, I think it's going to work, but for the precision issues.
> > > > Catch:
>
> No response? Does anyone have any insights as to the real parser policies?
>
> The documented grammar would lead me to believe that "fixed format",
> literally, (*digits  >> '.' >> +digits) | (+digits  >> '.' >>
> *digits), in Qi phraseology, might be possible. However, I am finding
> potentially this is not the case.
>
> > > After doing some background reading on floating point comparison ...
> > > it's not so simple as a==b, but I kind of knew that already. Even the
> > > simple absolute difference epsilon check is not so robust. I did a
> > > combination of nearly, relatively, essentially, etc, checks, and I
> > > think I have gotten past that immediate issue.
> > >
> > > Now I actually have a parsing issue... According to the Protobuf v2
> > > spec, this should parse, I think. However, it is failing to parse.
> > >
> > >   CHECK( parse_tried == expected_tried )
> > > with expansion:
> > >   false == true
> > > with message:
> > >   Source: syntax = 'proto2';option a =
> > > 933252761176835091419019014852578134454945897134240121039056292642745934626544776679309792643426912681734775655456099849022472225507009671635655261635494179939591695082743337844489696175391517474159955183289891072214770215979051082304343590361421008538352849939802258131368544983961387362613078430270357504.000000;
> > >   REQUIRE( tried )
> > > with expansion:
> > >   false
> > > with message:
> > >   Source: syntax = 'proto2';option a =
> > > 933252761176835091419019014852578134454945897134240121039056292642745934626544776679309792643426912681734775655456099849022472225507009671635655261635494179939591695082743337844489696175391517474159955183289891072214770215979051082304343590361421008538352849939802258131368544983961387362613078430270357504.000000;
> >
> > This much I can say at the moment, it does not appear to be a max
> > double issue, literally:
> >
> > 179769313486231570814527423731704356798070567525844996598917476803157260780028538760589558632766878171540458953514382464234321326889464182768467546703537516986049910576551282076245490090389328944075868508455133942304583236903222948165808559332123348274797826204144723168738177180919299881250404026184124858368.0000000000000000
> >
> > That would stand to reason, though, since my test data is randomly
> > generated within max double boundaries.
> >
> > I am scratching my head on the strict real policies, however. If the
> > docs are accurate, I do not see why that would not accept the fixed
> > double format...
> >
> > But it could be a concern that parsing is done apart from Qi rules. It
> > seems like some rigid logic embedded there.
> >
> > https://www.boost.org/doc/libs/1_68_0/libs/spirit/doc/html/spirit/qi/reference/numeric/real.html
> >
> > > >   REQUIRE( actual == expected )
> > > > with expansion:
> > > >   struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
> > > > 'opt_sign': null }
> > > >   ==
> > > >   struct my::protobuf::ast::floating_point_t { 'val': 1.16624e+306,
> > > > 'opt_sign': null }
> > > > with messages:
> > > >   Source: syntax = 'proto2';option L = 1.16624e+306;
> > > >   Delta was: 2.35309e+300
> > > >
> > > > There was clearly a delta involved, most like a precision issue
> > > > encountered during the parse. Without going into a great deal of
> > > > depth, if possible:
> > > >
> > > > namespace my { namespace protobuf { namespace ast {
> > > > using float_t = double;
> > > > enum num_sign_t {
> > > >     sign_none
> > > >     /// '-'
> > > >     , sign_minus
> > > >     /// '+'
> > > >     , sign_plus
> > > > };
> > > >
> > > > // Which includes
> > > > struct floating_point_t : numeric_t<float_t> { /* ... */ };
> > > > } } }
> > > >
> > > > // Without sign, dot required.
> > > > // >>> TODO: TBD: Precision issue? Or an actual double parsing issue? <<<
> > > > float_lit %= real_parser<ast::float_t, strict_ureal_policies<ast::float_t>>{};
> > > > floating_point_lit %= -num_lit_sign >> float_lit;
> > > >
> > > > qi::rule<It, ast::num_sign_t()> num_lit_sign;
> > > >
> > > > qi::rule<It, ast::float_t()> float_lit;
> > > > qi::rule<It, ast::floating_point_t()> floating_point_lit;
> > > >
> > > > https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#floating-point-literals
> > > > https://developers.google.com/protocol-buffers/docs/reference/proto2-spec#constant
> > > > https://www.boost.org/doc/libs/1_68_0/libs/spirit/doc/html/spirit/qi/reference/numeric/real.html
> > > >
> > > > Cheers,
> > > >
> > > > Michael Powell
_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users