[serialization]Nans and infinity in wide xml archive

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[serialization]Nans and infinity in wide xml archive

Elizabeta
Hi
I am using wide xml boost serialization archive. Because of the problem with loading of nans and infinities I tried the nonfinite facet as in the code snippet. This works, expect
it messes with the encoding of the xml arhive which is no longer UTF-8 like before the using the facet, and I believe it is because of the no_codecvt flag which is passed to the archive. But without this flag the loading of nans/infinity does not work. How to use the facet and keep the wanted UTF-8 encoding ?

    float d = std::numeric_limits<float>::quiet_NaN;
    {
        std::wofstream oss("C:\\test.xml");
        std::locale locale1(oss.getloc(), new boost::math::nonfinite_num_put<wchar_t>);
     
        oss.imbue(locale1);
        xml_woarchive oar(oss, no_codecvt);
        oar << BOOST_SERIALIZATION_NVP(d);
    }

    {
        float d;
        std::wifstream iss("C:\\test.xml");      
        std::locale locale1(iss.getloc(), new boost::math::nonfinite_num_get<wchar_t>);
        iss.imbue(locale1);
        xml_wiarchive iar(iss, no_codecvt);
        iar >> BOOST_SERIALIZATION_NVP(d);
        std::cout << d << std::endl;
    }
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [serialization]Nans and infinity in wide xml archive

Boost - Users mailing list
On 7/3/17 2:23 AM, Elizabeta via Boost-users wrote:

> Hi
> I am using wide xml boost serialization archive. Because of the problem with
> loading of nans and infinities I tried the nonfinite facet as in the code
> snippet. This works, expect
> it messes with the encoding of the xml arhive which is no longer UTF-8 like
> before the using the facet, and I believe it is because of the no_codecvt
> flag which is passed to the archive. But without this flag the loading of
> nans/infinity does not work. How to use the facet and keep the wanted UTF-8
> encoding ?
>
>      float d = std::numeric_limits<float>::quiet_NaN;
>      {
>          std::wofstream oss("C:\\test.xml");
>          std::locale locale1(oss.getloc(), new
> boost::math::nonfinite_num_put<wchar_t>);
>      
>          oss.imbue(locale1);
>          xml_woarchive oar(oss, no_codecvt);
>          oar << BOOST_SERIALIZATION_NVP(d);
>      }
>
>      {
>          float d;
>          std::wifstream iss("C:\\test.xml");
>          std::locale locale1(iss.getloc(), new
> boost::math::nonfinite_num_get<wchar_t>);
>          iss.imbue(locale1);
>          xml_wiarchive iar(iss, no_codecvt);
>          iar >> BOOST_SERIALIZATION_NVP(d);
>          std::cout << d << std::endl;
>      }
>

This is an old problem going back many years.  The problem isn't really
the serialization library, but rather writing and reading NaN values to
a text stream.  There have been solutions proposed, but there hasn't
been enough motivation pursue it.  These solution address the issue at
the stream level and don't touch serialization.  For more information it
would be best to search the mailing list - going back many years.

Note that this problem occurs only with stream/text i/o.  Binary formats
don't suffer from this - but they are, of course, do not create/use
portable archives.

One expedient way to address this at the serialization level is to step
back and ask oneself why he is serializing NaN values in the first
place.  They generally contain no useful information.  One could catch
them on output and replace them with a custom flag or boost.optional or
some ad hoc solution.  Or one could fix the NaN text i/o issue at a
lower level.

Robert Ramey

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [serialization]Nans and infinity in wide xml archive

Elizabeta
This post was updated on .
Hi Robert
I did template specialization of basic_text_iprimitive::load for double, and it works with visual studio 2010 compiler. Do you see something problematic with this code

namespace boost
{
    namespace archive
    {
        template<> template<>
        void basic_text_iprimitive< std::basic_istream<wchar_t, std::char_traits<wchar_t>> >::load<double>(double& t)
        {
            if (!is.fail())
            {
                std::wstring s;
                getline(is, s, L'<');
                is.putback(L'<');

                int index = s.find(L"IND");
                if (index != -1)
                {
                    t = std::numeric_limits<double>::quiet_NaN();
                    return;
                }

                index = s.find(L"NAN");
                if (index != -1)
                {
                    t = std::numeric_limits<double>::quiet_NaN();
                    return;
                }

                index = s.find(L"INF");
                if (index != -1)
                {
                    t = std::numeric_limits<double>::infinity();
                    return;
                }

                t = std::stod(s);
                return;
            }

            boost::serialization::throw_exception(
                archive_exception(archive_exception::input_stream_error)
                );
        }
    }
}
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [serialization]Nans and infinity in wide xml archive

Boost - Users mailing list
On 7/4/17 2:48 AM, Elizabeta via Boost-users wrote:

> Hi Robert
> I did template specialization of basic_text_iprimitive::load for double, and
> it works with visual studio 2010 compiler. Do you see something problematic
> with this code
>
> namespace boost {
>      namespace archive {
>          template<> template<>
>          void basic_text_iprimitive< std::basic_istream<wchar_t,
> std::char_traits&lt;wchar_t>> >::load<double>(double& t)
>          {
>              if (!is.fail())
>              {
>                  std::wstring s;
>                  getline(is, s, L'<');
>                  is.putback(L'<');
>
>                  int index = s.find(L"IND");
>                  if (index != -1)
>                  {
>                      t = std::numeric_limits<double>::quiet_NaN();
>                      return;
>                  }
>
>                  index = s.find(L"NAN");
>                  if (index != -1)
>                  {
>                      t = std::numeric_limits<double>::quiet_NaN();
>                      return;
>                  }
>
>                  index = s.find(L"INF");
>                  if (index != -1)
>                  {
>                      t = std::numeric_limits<double>::infinity();
>                      return;
>                  }
>
>                  t = std::stod(s);
>                  return;
>              }
>
>              boost::serialization::throw_exception(
>                  archive_exception(archive_exception::input_stream_error)
>                  );
>          }
> }
> }

Hmmm - very clever.  I have to say the specialization of just one member
function is something that would never have occurred to me.

I would wonder though.  I'm not sure that the Nan literals are standard.
  I also don't trust them to be portable.  Also to me it's a problem
with stream i/o rather than serialization itself.  So my instinct would
have been to maybe make a custom manipulator or something like that. But
still, you've got a solution which looks like it will work well for your
needs so good luck with this.

Robert Ramey

_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [serialization]Nans and infinity in wide xml archive

Boost - Users mailing list


> -----Original Message-----
> From: Boost-users [mailto:[hidden email]] On Behalf Of Robert Ramey via Boost-users
> Sent: 04 July 2017 16:05
> To: Elizabeta via Boost-users
> Cc: Robert Ramey
> Subject: Re: [Boost-users] [serialization]Nans and infinity in wide xml archive
>
> On 7/4/17 2:48 AM, Elizabeta via Boost-users wrote:
> > Hi Robert
> > I did template specialization of basic_text_iprimitive::load for double, and
> > it works with visual studio 2010 compiler. Do you see something problematic
> > with this code
> >
> > namespace boost {
> >      namespace archive {
> >          template<> template<>
> >          void basic_text_iprimitive< std::basic_istream<wchar_t,
> > std::char_traits&lt;wchar_t>> >::load<double>(double& t)
> >          {
> >              if (!is.fail())
> >              {
> >                  std::wstring s;
> >                  getline(is, s, L'<');
> >                  is.putback(L'<');
> >
> >                  int index = s.find(L"IND");
> >                  if (index != -1)
> >                  {
> >                      t = std::numeric_limits<double>::quiet_NaN();
> >                      return;
> >                  }
> >
> >                  index = s.find(L"NAN");
> >                  if (index != -1)
> >                  {
> >                      t = std::numeric_limits<double>::quiet_NaN();
> >                      return;
> >                  }
> >
> >                  index = s.find(L"INF");
> >                  if (index != -1)
> >                  {
> >                      t = std::numeric_limits<double>::infinity();
> >                      return;
> >                  }
> >
> >                  t = std::stod(s);
> >                  return;
> >              }
> >
> >              boost::serialization::throw_exception(
> >                  archive_exception(archive_exception::input_stream_error)
> >                  );
> >          }
> > }
> > }
>
> Hmmm - very clever.  I have to say the specialization of just one member
> function is something that would never have occurred to me.
>
> I would wonder though.  I'm not sure that the Nan literals are standard.
>   I also don't trust them to be portable.  Also to me it's a problem
> with stream i/o rather than serialization itself.  So my instinct would
> have been to maybe make a custom manipulator or something like that. But
> still, you've got a solution which looks like it will work well for your
> needs so good luck with this.

http://www.boost.org/doc/libs/1_64_0/libs/math/doc/html/math_toolkit/fp_facets/facets_intro.html

has some info on this:

"C99 standard for output of infinity and NaN

The C99 standard does specify how infinity and NaN are formatted by printf and similar output functions, and parsed by scanf and
similar input functions.

The following string representations are used:

Table 2.1. C99 Representation of Infinity and NaN

number

string
Positive infinity
"inf" or "infinity"
Positive NaN
"nan" or "nan(...)"
Negative infinity
"-inf" or "-infinity"
Negative NaN
"-nan" or "-nan(...)"

So following C99 provides a sensible 'standard' way of handling input and output of nonfinites in C++, and this implementation
follows most of these formats."

And Elizabeta's solution is close to this.  Using lower case might become more 'Standard'.

I don't know of any progress in a 'Standard' representation in C++14, 17 ....

HTH

Paul

---
Paul A. Bristow
Prizet Farmhouse
Kendal UK LA8 8AB
+44 (0) 1539 561830




_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [serialization]Nans and infinity in wide xml archive

Boost - Users mailing list
On 7/4/17 10:01 AM, Paul A. Bristow via Boost-users wrote:

>> I would wonder though.  I'm not sure that the Nan literals are standard.
>>    I also don't trust them to be portable.  Also to me it's a problem
>> with stream i/o rather than serialization itself.  So my instinct would
>> have been to maybe make a custom manipulator or something like that. But
>> still, you've got a solution which looks like it will work well for your
>> needs so good luck with this.
>
> http://www.boost.org/doc/libs/1_64_0/libs/math/doc/html/math_toolkit/fp_facets/facets_intro.html
>
> has some info on this:
>
> "C99 standard for output of infinity and NaN
>
> The C99 standard does specify how infinity and NaN are formatted by printf and similar output functions, and parsed by scanf and
> similar input functions.
>
> The following string representations are used:
>
> Table 2.1. C99 Representation of Infinity and NaN
>
> number
>
> string
> Positive infinity
> "inf" or "infinity"
> Positive NaN
> "nan" or "nan(...)"
> Negative infinity
> "-inf" or "-infinity"
> Negative NaN
> "-nan" or "-nan(...)"
>
> So following C99 provides a sensible 'standard' way of handling input and output of nonfinites in C++, and this implementation
> follows most of these formats."
>
> And Elizabeta's solution is close to this.  Using lower case might become more 'Standard'.
>
> I don't know of any progress in a 'Standard' representation in C++14, 17 ....
>
> HTH
>

This is all very interesting!  I know some time ago when this problem
first came up (I'm always getting the blame for this!)  There was some
work in addressing it.  I didn't take part because I was sort of burned
out on working on the serialization library.  But I would think it would
be a good thing if this could be "resolved" via a combination of
documentation and code so provide a canonical C++ solution which would
be compatible with just C++ stream i/o and hence automatically apply to
the the serialization library.

Robert Ramey


_______________________________________________
Boost-users mailing list
[hidden email]
https://lists.boost.org/mailman/listinfo.cgi/boost-users
Loading...