x3 semantic action and attribute transformation

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

x3 semantic action and attribute transformation

Olaf Peter
Hi,

I try in x3 to parse integers which can have seperators '_'. But it
fails to compile:

          detail::move_to(std::move(src), dest
          ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
            , typename attribute_category<Dest>::type());
            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
etc.

The idea behind is to parse it first into vector<int> and use these rule
with a semantic action (SA) with a 2nd rule to combine it finally to an
int. Append the SA directly to the 1st rule also fails to compile.

What is the correct way? Are there any simplifications also? The first
question seems to me fundamental to be solved.

BTW, why the syntax :
    auto const integer_def = integer %=
is there a better one?

Thanks,
Olaf

---8<---
#include <iosfwd>
//#define BOOST_SPIRIT_X3_DEBUG

#include <boost/spirit/home/x3.hpp>

#include <iostream>
#include <vector>
#include <algorithm> // for copy
#include <iterator>  // for ostream_iterator
#include <sstream>

namespace parser {

   namespace x3 = boost::spirit::x3;
   namespace iso8859_1 = boost::spirit::x3::iso8859_1;

   using iso8859_1::char_;

   typedef x3::rule<struct integer_class, uint> integer_type;
   typedef x3::rule<struct inner_integer_class, std::vector<int>>
inner_integer_type;

   integer_type const integer { "integer" };
   inner_integer_type const inner_integer { "inner integer" };


   auto combine_to_uint = [](auto &ctx) {
     auto const &v = x3::_val(ctx);
     std::ostringstream ss;
     std::copy(v.begin(), v.end(), std::ostream_iterator<int>(ss, ""));
     uint int_ { 0 };
     x3::_pass(ctx) = x3::parse(ss.str().begin(), ss.str().end(),
x3::uint_, int_);
     std::cout << int_ << '\n';
     x3::_attr(ctx) = int_;
   };

   x3::uint_parser<int, 10, 1, 1>  const digit = { };

   // Parse '1_000' as 1000
   auto const inner_integer_def =
     digit >> *( x3::omit[ -char_('_') ] >> digit )
     ;

   auto const integer_def = integer %= inner_integer[combine_to_uint];

   BOOST_SPIRIT_DEFINE(inner_integer, integer)
}


int main()
{
   namespace x3 = boost::spirit::x3;

   std::vector<std::string> const test_cases {
       "0",
       "1",
       "01",
       "1_000",
       "42_666_4711",
       "4_294_967_295", // uint32::max
       "4_294_967_296" // ??? how to get message about UINT_MAX failed??
       };

   typedef std::string::const_iterator iterator_type;

   for(auto str: test_cases) {
     iterator_type iter = str.begin();
     iterator_type const end = str.end();

     auto& rule = parser::integer;

     std::cout << "parse `" << str << "´:\n";

     std::vector<int> i;

     bool r = x3::phrase_parse(iter, end, rule, x3::space, i);

     if (r && iter == end) {
       std::cout << "succeeded\n";
     } else {
       std::cout << "failed\n";
     }
   }

   return 0;
}
--->8---

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: x3 semantic action and attribute transformation

Olaf Peter

> I try in x3 to parse integers which can have seperators '_'. But it
> fails to compile:
>
>           detail::move_to(std::move(src), dest
>           ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
>             , typename attribute_category<Dest>::type());
>             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> etc.
>
> The idea behind is to parse it first into vector<int> and use these rule
> with a semantic action (SA) with a 2nd rule to combine it finally to an
> int. Append the SA directly to the 1st rule also fails to compile.
>
> What is the correct way? Are there any simplifications also? The first
> question seems to me fundamental to be solved.
>
> BTW, why the syntax :
>     auto const integer_def = integer %=
> is there a better one?
>
> Thanks,
> Olaf
>
> ---8<---
> #include <iosfwd>
> //#define BOOST_SPIRIT_X3_DEBUG
>
> #include <boost/spirit/home/x3.hpp>
>
> #include <iostream>
> #include <vector>
> #include <algorithm> // for copy
> #include <iterator>  // for ostream_iterator
> #include <sstream>
>
> namespace parser {
>
>    namespace x3 = boost::spirit::x3;
>    namespace iso8859_1 = boost::spirit::x3::iso8859_1;
>
>    using iso8859_1::char_;
>
>    typedef x3::rule<struct integer_class, uint> integer_type;
>    typedef x3::rule<struct inner_integer_class, std::vector<int>>
> inner_integer_type;
>
>    integer_type const integer { "integer" };
>    inner_integer_type const inner_integer { "inner integer" };
>
>
>    auto combine_to_uint = [](auto &ctx) {
>      auto const &v = x3::_val(ctx);
>      std::ostringstream ss;
>      std::copy(v.begin(), v.end(), std::ostream_iterator<int>(ss, ""));
>      uint int_ { 0 };
>      x3::_pass(ctx) = x3::parse(ss.str().begin(), ss.str().end(),
> x3::uint_, int_);
>      std::cout << int_ << '\n';
>      x3::_attr(ctx) = int_;
>    };
>
>    x3::uint_parser<int, 10, 1, 1>  const digit = { };
>
>    // Parse '1_000' as 1000
>    auto const inner_integer_def =
>      digit >> *( x3::omit[ -char_('_') ] >> digit )
>      ;
>
>    auto const integer_def = integer %= inner_integer[combine_to_uint];
>
>    BOOST_SPIRIT_DEFINE(inner_integer, integer)
> }
>
>
> int main()
> {
>    namespace x3 = boost::spirit::x3;
>
>    std::vector<std::string> const test_cases {
>        "0",
>        "1",
>        "01",
>        "1_000",
>        "42_666_4711",
>        "4_294_967_295", // uint32::max
>        "4_294_967_296" // ??? how to get message about UINT_MAX failed??
>        };
>
>    typedef std::string::const_iterator iterator_type;
>
>    for(auto str: test_cases) {
>      iterator_type iter = str.begin();
>      iterator_type const end = str.end();
>
>      auto& rule = parser::integer;
>
>      std::cout << "parse `" << str << "´:\n";
>
>      std::vector<int> i;
>
>      bool r = x3::phrase_parse(iter, end, rule, x3::space, i);
>
>      if (r && iter == end) {
>        std::cout << "succeeded\n";
>      } else {
>        std::cout << "failed\n";
>      }
>    }
>
>    return 0;
> }
> --->8---

Probably I miss the hook to inform spirit x3 how to move src -> dest.
Simply x3::_val(ctx) = std::move(int_) doesn't solve it:

x3/support/traits/move_to.hpp:
...detail::move_to_plain ...  [with Source = std::vector<int>; Dest =
unsigned int, ...
...detail::move_to...[with Source = std::vector<int>; Dest = unsigned
int, ...


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: x3 semantic action and attribute transformation

Matthijs Möhlmann
On 3/8/17 2:21 PM, Olaf Peter wrote:

>> I try in x3 to parse integers which can have seperators '_'. But it
>> fails to compile:
>>
>>           detail::move_to(std::move(src), dest
>>           ~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~
>>             , typename attribute_category<Dest>::type());
>>             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> etc.
>>
>> The idea behind is to parse it first into vector<int> and use these rule
>> with a semantic action (SA) with a 2nd rule to combine it finally to an
>> int. Append the SA directly to the 1st rule also fails to compile.
>>
>> What is the correct way? Are there any simplifications also? The first
>> question seems to me fundamental to be solved.
>>
>> BTW, why the syntax :
>>     auto const integer_def = integer %=
>> is there a better one?
>>
>> Thanks,
>> Olaf
>>
>> ---8<---
>> #include <iosfwd>
>> //#define BOOST_SPIRIT_X3_DEBUG
>>
>> #include <boost/spirit/home/x3.hpp>
>>
>> #include <iostream>
>> #include <vector>
>> #include <algorithm> // for copy
>> #include <iterator>  // for ostream_iterator
>> #include <sstream>
>>
>> namespace parser {
>>
>>    namespace x3 = boost::spirit::x3;
>>    namespace iso8859_1 = boost::spirit::x3::iso8859_1;
>>
>>    using iso8859_1::char_;
>>
>>    typedef x3::rule<struct integer_class, uint> integer_type;
>>    typedef x3::rule<struct inner_integer_class, std::vector<int>>
>> inner_integer_type;
>>
>>    integer_type const integer { "integer" };
>>    inner_integer_type const inner_integer { "inner integer" };
>>
>>
>>    auto combine_to_uint = [](auto &ctx) {
>>      auto const &v = x3::_val(ctx);
>>      std::ostringstream ss;
>>      std::copy(v.begin(), v.end(), std::ostream_iterator<int>(ss, ""));
>>      uint int_ { 0 };
>>      x3::_pass(ctx) = x3::parse(ss.str().begin(), ss.str().end(),
>> x3::uint_, int_);
>>      std::cout << int_ << '\n';
>>      x3::_attr(ctx) = int_;
>>    };
>>
>>    x3::uint_parser<int, 10, 1, 1>  const digit = { };
>>
>>    // Parse '1_000' as 1000
>>    auto const inner_integer_def =
>>      digit >> *( x3::omit[ -char_('_') ] >> digit )
>>      ;
>>
>>    auto const integer_def = integer %= inner_integer[combine_to_uint];
>>
>>    BOOST_SPIRIT_DEFINE(inner_integer, integer)
>> }
>>
>>
>> int main()
>> {
>>    namespace x3 = boost::spirit::x3;
>>
>>    std::vector<std::string> const test_cases {
>>        "0",
>>        "1",
>>        "01",
>>        "1_000",
>>        "42_666_4711",
>>        "4_294_967_295", // uint32::max
>>        "4_294_967_296" // ??? how to get message about UINT_MAX failed??
>>        };
>>
>>    typedef std::string::const_iterator iterator_type;
>>
>>    for(auto str: test_cases) {
>>      iterator_type iter = str.begin();
>>      iterator_type const end = str.end();
>>
>>      auto& rule = parser::integer;
>>
>>      std::cout << "parse `" << str << "´:\n";
>>
>>      std::vector<int> i;
>>
>>      bool r = x3::phrase_parse(iter, end, rule, x3::space, i);
>>
>>      if (r && iter == end) {
>>        std::cout << "succeeded\n";
>>      } else {
>>        std::cout << "failed\n";
>>      }
>>    }
>>
>>    return 0;
>> }
>> --->8---
> Probably I miss the hook to inform spirit x3 how to move src -> dest.
> Simply x3::_val(ctx) = std::move(int_) doesn't solve it:
>
> x3/support/traits/move_to.hpp:
> ...detail::move_to_plain ...  [with Source = std::vector<int>; Dest =
> unsigned int, ...
> ...detail::move_to...[with Source = std::vector<int>; Dest = unsigned
> int, ...
>

Perhaps you can use a skipper, then skip the '_' and let x3 do the work.

In <boost_root>/libs/spirit/test/x3/skip.cpp is an example how to use that.

Hopefully that will help.

Regards,

Matthijs

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: x3 semantic action and attribute transformation

sehe
In reply to this post by Olaf Peter
On 08-03-17 10:35, Olaf Peter wrote:
> BTW, why the syntax :
>     auto const integer_def = integer %=
> is there a better one?

Yes, use the third argument to the rule:

   19     typedef x3::rule<struct integer_class, uint, true>
integer_type;                                                                                              


I think your sample is a bit complicated. I'd suggest something like:
http://coliru.stacked-crooked.com/a/df8986cf07f77146

You could do the overflow check by checking each iteration in the loop
makes `r` bigger. If it wraps around there was an overflow.

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: x3 semantic action and attribute transformation

sehe
In reply to this post by Olaf Peter
On 08-03-17 10:35, Olaf Peter wrote:

>    auto combine_to_uint = [](auto &ctx) {
>      auto const &v = x3::_val(ctx);
>      std::ostringstream ss;
>      std::copy(v.begin(), v.end(), std::ostream_iterator<int>(ss, ""));
>      uint int_ { 0 };
>      x3::_pass(ctx) = x3::parse(ss.str().begin(), ss.str().end(),
> x3::uint_, int_);
>      std::cout << int_ << '\n';
>      x3::_attr(ctx) = int_;
>    };

Here it seems you mainly confused `_attr` and `_val`. Avoid confusing
names like `int_`.

Here's your own sample "fixed" (I still think it's too complicated):
http://coliru.stacked-crooked.com/a/a6e053b5f552f766


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: x3 semantic action and attribute transformation

sehe
In reply to this post by Matthijs Möhlmann
On 08-03-17 15:12, Matthijs Möhlmann wrote:

      
Perhaps you can use a skipper, then skip the '_' and let x3 do the work.

You can't because `int_parser` is a lexeme


------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: x3 semantic action and attribute transformation

Olaf Peter
In reply to this post by sehe
Hello Seth,

thank you much!

> On 08-03-17 10:35, Olaf Peter wrote:
>>    auto combine_to_uint = [](auto &ctx) {
>>      auto const &v = x3::_val(ctx);
>>      std::ostringstream ss;
>>      std::copy(v.begin(), v.end(), std::ostream_iterator<int>(ss, ""));
>>      uint int_ { 0 };
>>      x3::_pass(ctx) = x3::parse(ss.str().begin(), ss.str().end(),
>> x3::uint_, int_);
>>      std::cout << int_ << '\n';
>>      x3::_attr(ctx) = int_;
>>    };
>
> Here it seems you mainly confused `_attr` and `_val`. Avoid confusing
> names like `int_`.

maybe it was left from prior test; _attr is the attribute from rule and
_val from lexeme which calls the semantic action, isn't it?

> Here's your own sample "fixed" (I still think it's too complicated):
> http://coliru.stacked-crooked.com/a/a6e053b5f552f766

thank you for this approach, I did try to solve the overflow problem and
come to this solution attached (for some reason, stacked-crooked doesn't
share my code), which works so far but looks quite complicated.

Also, I changed the rule to be more restrict, but this requires changes
in the 'combine' lambda function. These are required to cover the last 4
test cases which aren't valid ints.

Thanks,
Olaf

---8<---
#include <iosfwd>
//#define BOOST_SPIRIT_X3_DEBUG

#include <boost/spirit/home/x3.hpp>

#include <iostream>


namespace parser {

   namespace x3 = boost::spirit::x3;

   using x3::char_;

   typedef x3::rule<struct integer_class, uint32_t> integer_type;

   integer_type const integer { "integer" };

   constexpr uint32_t digit_treshold(auto i) {
       return std::numeric_limits<decltype(i)>::max() % 10;
   }

   auto const combine = [](auto &ctx) {
       //typedef decltype(x3::_attr(ctx)) base_type; // =>> templated?
       typedef uint32_t base_type;
       base_type result { 0 };
       uint32_t iter_cnt { 0 };
       for (auto&& ch : x3::_attr(ctx)) {
           switch (ch) {
               case '_': break;
               default:
                   if(iter_cnt > std::numeric_limits<base_type>::digits10) {
                       x3::_pass(ctx) = false;
                       continue;
                   }
                   if(iter_cnt++ <
std::numeric_limits<base_type>::digits10) {
                       result = result*10 + (ch - '0');
                   }
                   else {
#if 0
                       switch(ch - '0') {
                       case 0: // [[fallthrough]]
                       case 1: // [[fallthrough]]
                       case 2: // [[fallthrough]]
                       case 3: // [[fallthrough]]
                       case 4: // [[fallthrough]]
                       case 5:
                           result = result*10 + (ch - '0');
                           break;
                       default:
                           x3::_pass(ctx) = false;
                       }
#else
                       if((unsigned)(ch - '0') > digit_treshold(result)) {
                       x3::_pass(ctx) = false;
                       }
                       else {
                       result = result*10 + (ch - '0');
                       }
#endif
                   }
           }
       }

       x3::_val(ctx) = result;
   };

   // Parse '1_000' as 1000
   auto const integer_def =
     //x3::lexeme [ char_("0-9") >> *(char_("0-9_")) ] [combine] /*
_val(ctx) => uint */
     x3::lexeme [ +char_("0-9_") ] [combine]
     ;

   BOOST_SPIRIT_DEFINE(integer)
}


int main()
{
   namespace x3 = boost::spirit::x3;

   std::vector<std::string> const test_cases {
       "0",
       "1",
       "01",
       "1_000",
       "42_666_4711",
       "4_294_967_295", // uint32::max
       // below must fail to parse
       "4_294_967_296",
       "4_294_967_295_0",
       "4_294_967_295_00",
       "_42", "42_",
       " 4 2 ", "4 _2"
       };

   typedef std::string::const_iterator iterator_type;

   for(auto str: test_cases) {
     iterator_type iter = str.begin();
     iterator_type const end = str.end();

     auto& rule = parser::integer;

     std::cout << "parse `" << str << "´:\n";

     uint32_t i;

     bool r = x3::phrase_parse(iter, end, rule, x3::space, i);

     if (r && iter == end) {
       std::cout << "succeeded: \"" << str << "\" -> " << i << "\n";
     } else {
       std::cout << "failed\n";
     }
   }

   return 0;
}

--->8---

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: x3 semantic action and attribute transformation

Olaf Peter

>> Here's your own sample "fixed" (I still think it's too complicated):
>> http://coliru.stacked-crooked.com/a/a6e053b5f552f766
>
> thank you for this approach, I did try to solve the overflow problem and
> come to this solution attached (for some reason, stacked-crooked doesn't
> share my code), which works so far but looks quite complicated.

The idea behind was that it shall work for uint{64,128} too. Normally a
promoted type and compare against UMAX is enough.

------------------------------------------------------------------------------
Announcing the Oxford Dictionaries API! The API offers world-renowned
dictionary content that is easy and intuitive to access. Sign up for an
account today to start using our lexical data to power your apps and
projects. Get started today and enter our developer competition.
http://sdm.link/oxford
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Loading...