bug in "double" parser causes program abort (minimum example provided)

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

bug in "double" parser causes program abort (minimum example provided)

Mathias Born
Hi,

The qi::double_ parser doesn't handle values outside the double range
well.
For example, trying to parse "1e3000" or "1e-3000" triggers an assert
and aborts the program.
The attached test program, built with VS2013 on Windows, says:

Assertion failed: dim < sizeof(exponents)/sizeof(double), file c:\__cpp__\boost\boost\spirit\home\support\detail\pow10.hpp, line 87

There isn't an obvious way to deal with this, but aborting the
program is further down the list.
Values which are too small, like 1e-3000, may just be returned
as zero, while values which are too large could be returned
as NaN, infinity or std::numeric_limits<>::max(), or could
throw an exception. Would be nice if this could be configured
somehow.

Best Regards,
Mathias
------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

case_double.cpp (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bug in "double" parser causes program abort (minimum example provided)

Joel de Guzman
On 10/15/14, 5:18 AM, [hidden email] wrote:

> Hi,
>
> The qi::double_ parser doesn't handle values outside the double range
> well.
> For example, trying to parse "1e3000" or "1e-3000" triggers an assert
> and aborts the program.
> The attached test program, built with VS2013 on Windows, says:
>
> Assertion failed: dim < sizeof(exponents)/sizeof(double), file c:\__cpp__\boost\boost\spirit\home\support\detail\pow10.hpp, line 87
>
> There isn't an obvious way to deal with this, but aborting the
> program is further down the list.
> Values which are too small, like 1e-3000, may just be returned
> as zero, while values which are too large could be returned
> as NaN, infinity or std::numeric_limits<>::max(), or could
> throw an exception. Would be nice if this could be configured
> somehow.

Indeed, that big exponent will not fit a double precision IEEE 754.
IEEE 754 supports a maximum (decimal) exponent of 308:

   http://en.wikipedia.org/wiki/IEEE_floating_point

I'm not sure how best to deal with this. Returning zero or infinity
is possible, but perhaps it's better to simply fail the parse like
we do when an integer overflows, for example. Thoughts?

Regards,
--
Joel de Guzman
http://www.ciere.com
http://boost-spirit.com
http://www.cycfi.com/


------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: bug in "double" parser causes program abort (minimum example provided)

Jeroen Habraken
On 14 October 2014 23:15, Joel de Guzman <[hidden email]> wrote:
On 10/15/14, 5:18 AM, [hidden email] wrote:
> Hi,
>
> The qi::double_ parser doesn't handle values outside the double range
> well.
> For example, trying to parse "1e3000" or "1e-3000" triggers an assert
> and aborts the program.
> The attached test program, built with VS2013 on Windows, says:
>
> Assertion failed: dim < sizeof(exponents)/sizeof(double), file c:\__cpp__\boost\boost\spirit\home\support\detail\pow10.hpp, line 87
>
> There isn't an obvious way to deal with this, but aborting the
> program is further down the list.
> Values which are too small, like 1e-3000, may just be returned
> as zero, while values which are too large could be returned
> as NaN, infinity or std::numeric_limits<>::max(), or could
> throw an exception. Would be nice if this could be configured
> somehow.

Indeed, that big exponent will not fit a double precision IEEE 754.
IEEE 754 supports a maximum (decimal) exponent of 308:

   http://en.wikipedia.org/wiki/IEEE_floating_point

I'm not sure how best to deal with this. Returning zero or infinity
is possible, but perhaps it's better to simply fail the parse like
we do when an integer overflows, for example. Thoughts?

Failing to parse is the way to go in my opinion, unless otherwise specified in the floating point policy of course, primarily due to the consistency with the integer parsers.
 
Regards,
--
Joel de Guzman
http://www.ciere.com
http://boost-spirit.com
http://www.cycfi.com/

Jeroen 

------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: bug in "double" parser causes program abort (minimum example provided)

Mathias Born
In reply to this post by Joel de Guzman
On 15.10.2014, 08:15:11 Joel de Guzman wrote:

> On 10/15/14, 5:18 AM, [hidden email] wrote:
>> Hi,
>>
>> The qi::double_ parser doesn't handle values outside the double range
>> well.
>> For example, trying to parse "1e3000" or "1e-3000" triggers an assert
>> and aborts the program.
>> The attached test program, built with VS2013 on Windows, says:
>>
>> Assertion failed: dim < sizeof(exponents)/sizeof(double), file c:\__cpp__\boost\boost\spirit\home\support\detail\pow10.hpp, line 87
>>
>> There isn't an obvious way to deal with this, but aborting the
>> program is further down the list.
>> Values which are too small, like 1e-3000, may just be returned
>> as zero, while values which are too large could be returned
>> as NaN, infinity or std::numeric_limits<>::max(), or could
>> throw an exception. Would be nice if this could be configured
>> somehow.

> Indeed, that big exponent will not fit a double precision IEEE 754.
> IEEE 754 supports a maximum (decimal) exponent of 308:

>    http://en.wikipedia.org/wiki/IEEE_floating_point

> I'm not sure how best to deal with this. Returning zero or infinity
> is possible, but perhaps it's better to simply fail the parse like
> we do when an integer overflows, for example. Thoughts?

> Regards,

Ok, digging deeper uncovers more. The attached new test program
produces this output:

3ff0 0000 0000 0000   (1)
3ff0 0000 0000 0001   (1.0000000000000002)
3ff0 0000 0000 0000   (1.00000000000000001)
0000 0000 0000 0014   (0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001)
0000 0000 0000 0002   (0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001)
0000 0000 0000 0000   (0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001)
0000 0000 0000 0000   (0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001)
7f76 c8e5 ca23 9028   (1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)
7fac 7b1f 3cac 7432   (10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)
7fe1 ccf3 85eb c89f   (100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)
7ff0 0000 0000 0000   (1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)

Input with too many digits is truncated, values which are two large
are returned as infinity (last line). But no parse fail.
Thus, you actually do know how to deal with this :-)

I'd say this behavior is very useful, e.g. for parsing user input.
But it's certainly not the right thing for every application.
Controlling this via policies seems the way to go.
But please make it consistent :-)

Best Regards,
Mathias




------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

case_double.cpp (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: bug in "double" parser causes program abort (minimum example provided)

Joel de Guzman
On 10/16/14, 4:58 AM, [hidden email] wrote:

> On 15.10.2014, 08:15:11 Joel de Guzman wrote:
>> On 10/15/14, 5:18 AM, [hidden email] wrote:
>>> Hi,
>>>
>>> The qi::double_ parser doesn't handle values outside the double range
>>> well.
>>> For example, trying to parse "1e3000" or "1e-3000" triggers an assert
>>> and aborts the program.
>>> The attached test program, built with VS2013 on Windows, says:
>>>
>>> Assertion failed: dim < sizeof(exponents)/sizeof(double), file c:\__cpp__\boost\boost\spirit\home\support\detail\pow10.hpp, line 87
>>>
>>> There isn't an obvious way to deal with this, but aborting the
>>> program is further down the list.
>>> Values which are too small, like 1e-3000, may just be returned
>>> as zero, while values which are too large could be returned
>>> as NaN, infinity or std::numeric_limits<>::max(), or could
>>> throw an exception. Would be nice if this could be configured
>>> somehow.
>
>> Indeed, that big exponent will not fit a double precision IEEE 754.
>> IEEE 754 supports a maximum (decimal) exponent of 308:
>
>>     http://en.wikipedia.org/wiki/IEEE_floating_point
>
>> I'm not sure how best to deal with this. Returning zero or infinity
>> is possible, but perhaps it's better to simply fail the parse like
>> we do when an integer overflows, for example. Thoughts?
>
>> Regards,
>
> Ok, digging deeper uncovers more. The attached new test program
> produces this output:
>
> 3ff0 0000 0000 0000   (1)
> 3ff0 0000 0000 0001   (1.0000000000000002)
> 3ff0 0000 0000 0000   (1.00000000000000001)
> 0000 0000 0000 0014   (0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001)
> 0000 0000 0000 0002   (0.00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001)
> 0000 0000 0000 0000   (0.000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001)
> 0000 0000 0000 0000   (0.0000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000001)
> 7f76 c8e5 ca23 9028   (1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)
> 7fac 7b1f 3cac 7432   (10000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)
> 7fe1 ccf3 85eb c89f   (100000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)
> 7ff0 0000 0000 0000   (1000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000)
>
> Input with too many digits is truncated, values which are two large
> are returned as infinity (last line). But no parse fail.
> Thus, you actually do know how to deal with this :-)
>
> I'd say this behavior is very useful, e.g. for parsing user input.
> But it's certainly not the right thing for every application.
> Controlling this via policies seems the way to go.
> But please make it consistent :-)

These are wonderful tests! Thank you for taking the time to do it!
I'll add it to the test cases.

Regards,
--
Joel de Guzman
http://www.ciere.com
http://boost-spirit.com
http://www.cycfi.com/


------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: bug in "double" parser causes program abort (minimum example provided)

Mathias Born
On 16.10.2014, 04:03:12 Joel de Guzman wrote:
> On 10/16/14, 4:58 AM, [hidden email] wrote:
>> On 15.10.2014, 08:15:11 Joel de Guzman wrote:
>>> On 10/15/14, 5:18 AM, [hidden email] wrote:
>> [...]

> These are wonderful tests! Thank you for taking the time to do it!
> I'll add it to the test cases.

> Regards,

I'm glad to be of help. There are certainly more borderline cases
to test, for example involving negative numbers.
And let us not loose sight of the fact that parsing "1e3000" is
still broken.

Best Regards,
Mathias





------------------------------------------------------------------------------
Comprehensive Server Monitoring with Site24x7.
Monitor 10 servers for $9/Month.
Get alerted through email, SMS, voice calls or mobile push notifications.
Take corrective actions from your mobile device.
http://p.sf.net/sfu/Zoho
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: bug in "double" parser causes program abort (minimum example provided)

Paul Harris-2
 <Mathias.Born <at> gmx.de> writes:

>
> On 16.10.2014, 04:03:12 Joel de Guzman wrote:
> > On 10/16/14, 4:58 AM, Mathias.Born <at> gmx.de wrote:
> >> On 15.10.2014, 08:15:11 Joel de Guzman wrote:
> >>> On 10/15/14, 5:18 AM, Mathias.Born <at> gmx.de wrote:
> >> [...]
>
> > These are wonderful tests! Thank you for taking the time to do it!
> > I'll add it to the test cases.
>
> > Regards,
>
> I'm glad to be of help. There are certainly more borderline cases
> to test, for example involving negative numbers.
> And let us not loose sight of the fact that parsing "1e3000" is
> still broken.
>
> Best Regards,
> Mathias
>

Any progress on this issue?
I just hit this and am wondering about a workaround, at least for now.

I will probably replace the assert with an exception throw, until a better
solution comes along.

The assert is in pow10_helper<>,
            BOOST_ASSERT(dim < sizeof(exponents)/sizeof(double));
You should be able to detect the problem before we get this deep,
but if we wait until we call pow10_helper, then we either have to throw (bad
for performance?) or I assume we have to change the API so it returns
true/false success flag, and return the value via a reference parameter.

That sort of restructuring probably affects a lot of spirit, which is why
I'll just go for an exception throw for now.

Thoughts?

cheers,
Paul



------------------------------------------------------------------------------
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: bug in "double" parser causes program abort (minimum example provided)

Joel de Guzman
On 11/5/14, 11:49 AM, Paul wrote:

>   <Mathias.Born <at> gmx.de> writes:
>
>>
>> On 16.10.2014, 04:03:12 Joel de Guzman wrote:
>>> On 10/16/14, 4:58 AM, Mathias.Born <at> gmx.de wrote:
>>>> On 15.10.2014, 08:15:11 Joel de Guzman wrote:
>>>>> On 10/15/14, 5:18 AM, Mathias.Born <at> gmx.de wrote:
>>>> [...]
>>
>>> These are wonderful tests! Thank you for taking the time to do it!
>>> I'll add it to the test cases.
>>
>>> Regards,
>>
>> I'm glad to be of help. There are certainly more borderline cases
>> to test, for example involving negative numbers.
>> And let us not loose sight of the fact that parsing "1e3000" is
>> still broken.
>>
>> Best Regards,
>> Mathias
>>
>
> Any progress on this issue?
> I just hit this and am wondering about a workaround, at least for now.
>
> I will probably replace the assert with an exception throw, until a better
> solution comes along.
>
> The assert is in pow10_helper<>,
>              BOOST_ASSERT(dim < sizeof(exponents)/sizeof(double));
> You should be able to detect the problem before we get this deep,
> but if we wait until we call pow10_helper, then we either have to throw (bad
> for performance?) or I assume we have to change the API so it returns
> true/false success flag, and return the value via a reference parameter.
>
> That sort of restructuring probably affects a lot of spirit, which is why
> I'll just go for an exception throw for now.

I'll make it a point to fix these issues ASAP. I'll work on it in the
coming days. Thanks for your patience!

Regards,
--
Joel de Guzman
http://www.ciere.com
http://boost-spirit.com
http://www.cycfi.com/


------------------------------------------------------------------------------
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: bug in "double" parser causes program abort (minimum example provided)

Joel de Guzman
On 11/5/14, 12:51 PM, Joel de Guzman wrote:

>> Any progress on this issue?
>> I just hit this and am wondering about a workaround, at least for now.
>>
>> I will probably replace the assert with an exception throw, until a better
>> solution comes along.
>>
>> The assert is in pow10_helper<>,
>>               BOOST_ASSERT(dim < sizeof(exponents)/sizeof(double));
>> You should be able to detect the problem before we get this deep,
>> but if we wait until we call pow10_helper, then we either have to throw (bad
>> for performance?) or I assume we have to change the API so it returns
>> true/false success flag, and return the value via a reference parameter.
>>
>> That sort of restructuring probably affects a lot of spirit, which is why
>> I'll just go for an exception throw for now.
>
> I'll make it a point to fix these issues ASAP. I'll work on it in the
> coming days. Thanks for your patience!

This is now fixed in the develop branch along with a whole slew of
improvements in floating point precision parsing (the corner cases).
There are some backward incompatible changes, but it should only
affect those who are using the real parser policies, in patrticular,
those who specialize parse_frac_n. The changes will be documented in
due time.

Regards,
--
Joel de Guzman
http://www.ciere.com
http://boost-spirit.com
http://www.cycfi.com/


------------------------------------------------------------------------------
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: bug in "double" parser causes program abort (minimum example provided)

Paul Harris-2
Just wanted to say Thanks :)

On 7 November 2014 18:28, Joel de Guzman <[hidden email]> wrote:
On 11/5/14, 12:51 PM, Joel de Guzman wrote:

>> Any progress on this issue?
>> I just hit this and am wondering about a workaround, at least for now.
>>
>> I will probably replace the assert with an exception throw, until a better
>> solution comes along.
>>
>> The assert is in pow10_helper<>,
>>               BOOST_ASSERT(dim < sizeof(exponents)/sizeof(double));
>> You should be able to detect the problem before we get this deep,
>> but if we wait until we call pow10_helper, then we either have to throw (bad
>> for performance?) or I assume we have to change the API so it returns
>> true/false success flag, and return the value via a reference parameter.
>>
>> That sort of restructuring probably affects a lot of spirit, which is why
>> I'll just go for an exception throw for now.
>
> I'll make it a point to fix these issues ASAP. I'll work on it in the
> coming days. Thanks for your patience!

This is now fixed in the develop branch along with a whole slew of
improvements in floating point precision parsing (the corner cases).
There are some backward incompatible changes, but it should only
affect those who are using the real parser policies, in patrticular,
those who specialize parse_frac_n. The changes will be documented in
due time.

Regards,
--
Joel de Guzman
http://www.ciere.com
http://boost-spirit.com
http://www.cycfi.com/


------------------------------------------------------------------------------
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general


------------------------------------------------------------------------------
Download BIRT iHub F-Type - The Free Enterprise-Grade BIRT Server
from Actuate! Instantly Supercharge Your Business Reports and Dashboards
with Interactivity, Sharing, Native Excel Exports, App Integration & more
Get technology previously reserved for billion-dollar corporations, FREE
http://pubads.g.doubleclick.net/gampad/clk?id=157005751&iu=/4140/ostg.clktrk
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general