X3: Using dynamic Repetition Parser Directives

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

X3: Using dynamic Repetition Parser Directives

Mike Gresens
Hi,

Let's start with x3::repeat(3)[foo]. Ok, no problem.
Now let's say we want a rule like x3::repeat(n)[foo]. And n is parsed before.
Something like x3::int_ > x3::repeat(n)[foo].

How we can use the int parsed by x3::int_ as 'n' in x3::repeat(n)[foo] ?

Thanks,
Mike...
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

sehe
On 27-01-16 21:59, Mike Gresens wrote:

> Hi,
>
> Let's start with x3::repeat(3)[foo]. Ok, no problem.
> Now let's say we want a rule like x3::repeat(n)[foo]. And n is parsed
> before.
> Something like x3::int_ > x3::repeat(n)[foo].
>
> How we can use the int parsed by x3::int_ as 'n' in x3::repeat(n)[foo] ?
>
>

For now you can do this:
https://stackoverflow.com/questions/33624149/boost-spirit-x3-cannot-compile-repeat-directive-with-variable-factor/33627991#33627991

There was a previous discussion here:

 -
http://boost.2283326.n4.nabble.com/Porting-from-Qi-to-X3-td4676224.html#a4676425
 -
http://boost.2283326.n4.nabble.com/Porting-from-Qi-to-X3-td4676224.html#a4676613

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

Mike Gresens
Thanks a lot!

May i say that the solution looks terrible ugly?!
All other things in X3 look beautiful. But this...

Why not something like this:

#include <boost/spirit/home/x3.hpp>
#include <string>
#include <iostream>

namespace x3 = boost::spirit::x3;

struct n_;

int main()
{
	int n;
//	auto foo_rule = x3::with<n_>(std::ref(n))[ x3::omit [ x3::int_ [ ( [](auto& ctx){ x3::get<n_>(ctx).get() = x3::_attr(ctx); } ) ] ] > x3::repeat(n)[x3::char_] ]; // does not work
	auto foo_rule = x3::with<n_>(std::ref(n))[ x3::omit [ x3::int_ [ ( [](auto& ctx){ x3::get<n_>(ctx).get() = x3::_attr(ctx); } ) ] ] > x3::repeat(std::ref(n))[x3::char_] ];

	std::string foo("3 abc");
	std::string str;
	x3::phrase_parse(foo.begin(), foo.end(), foo_rule, x3::space, str);
	std::cout << str << std::endl;
	return 0;
}

Of course we get compiler errors here, but only because x3::repeat(integral_datatype) [...].
If we have x3::repeat(integral_datatype or reference_wrapper<integral_datatype>) [...] the code should work. But still ugly.

Mike...
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

sehe
On 28-01-16 19:39, Mike Gresens wrote:
> Of course we get compiler errors here, but only because
> x3::repeat(integral_datatype) [...].
> If we have x3::repeat(integral_datatype or
> reference_wrapper<integral_datatype>) [...] the code should work. But still
> ugly.
As the other links clearly show - something like it is in the works. I'm
not involved though, so you'll have to ask the others about it

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

Mike Gresens
In reply to this post by sehe
Hi sehe,

I tried the provided solution:

auto r = rule<struct _r, std::string > {}
                  %= with<_n>(std::ref(n))
                        [ omit[uint_[number] ] >> *(eps [more] >> char_) >> eps [done] ];

But the string remains empty.
And I remember something like: If you use semantic actions then you have to do it everywhere.

auto r = rule<struct _r, std::string > {}
                  %= with<_n>(std::ref(n))
                        [ omit[uint_[number] ] >> *(eps [more] >> char_ [push] ) >> eps [done] ];

So using an explicit push action does fill the string for me.

Mike...
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

sehe
On 31-01-16 13:24, Mike Gresens wrote:

> I tried the provided solution:
>
> auto r = rule<struct _r, std::string > {}
>                   %= with<_n>(std::ref(n))
>                         [ omit[uint_[number] ] >> *(eps [more] >> char_) >>
> eps [done] ];
>
> But the string remains empty.
> And I remember something like: If you use semantic actions then you have to
> do it everywhere.
>
> auto r = rule<struct _r, std::string > {}
>                   %= with<_n>(std::ref(n))
>                         [ omit[uint_[number] ] >> *(eps [more] >> char_
> [push] ) >> eps [done] ];
>
> So using an explicit push action does fill the string for me.
Is there a question?
You don't show the relevant code.

Why do you write `eps[done]`?

Lastly, the idiomatic way in X3 is to use the rule template argument:
rule<struct _r, std::string, true>

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

Mike Gresens
Was just the original code from

https://stackoverflow.com/questions/33624149/boost-spirit-x3-cannot-compile-repeat-directive-with-variable-factor/33627991#33627991

replacing std::vector<int> by std::string and x3::int_ by x3::char_

Would like to provide you buildable code, but currently this code seems to be completely broken:

#include <boost/spirit/home/x3.hpp>
#include <boost/spirit/home/x3/binary.hpp>
#include <string>
#include <memory>
#include <iostream>

namespace x3 = boost::spirit::x3;

namespace hessian {

typedef std::string string_t;

namespace parser {
namespace detail {

//string ::= s b1 b0 <utf8-data> string
//       ::= S b1 b0 <utf8-data>
//       ::= [x00-x1f] <utf8-data>

// NOTE: The length means number of UTF16 characters but the content is given in UTF8 characters!

struct length_tag;
const auto length_action = [](auto& ctx)
{
        *x3::get<length_tag>(ctx) = x3::_attr(ctx);
};
const auto more_action = [](auto &ctx)
{
        x3::_pass(ctx) = *x3::get<length_tag>(ctx) >  x3::_val(ctx).size();
};
const auto done_action = [](auto &ctx)
{
        x3::_pass(ctx) = *x3::get<length_tag>(ctx) == x3::_val(ctx).size();
};
const auto push_action = [](auto &ctx)
{
        x3::_val(ctx).push_back(x3::_attr(ctx));
};

const x3::rule<class string_rule, string_t> string_rule("string");
const x3::rule<class string1_rule, string_t> string1_rule;
const x3::rule<class string2_rule, string_t> string2_rule;

const auto length_rule = x3::omit[x3::big_word [length_action] ];
const auto content_rule = x3::lexeme[ *(x3::eps [more_action] >> x3::char_ [push_action]) >> x3::eps [done_action] ];

const auto string_rule_def = x3::with<length_tag>(std::make_shared<std::size_t>())[string1_rule | string2_rule];
const auto string1_rule_def = x3::lit('S') >> length_rule >> content_rule;
const auto string2_rule_def = x3::lit('s') >> length_rule >> content_rule >> string_rule;

BOOST_SPIRIT_DEFINE(string_rule, string1_rule, string2_rule);

}

using detail::string_rule;

}
}

void test_string()
{
        const std::string text("s\x00\x07hello, S\x00\x05world"s);
        try
        {
        hessian::string_t attr;
        std::cout << x3::parse(text.begin(), text.end(), x3::eps > hessian::parser::string_rule, attr) << std::endl;
        std::cout << attr << std::endl;
        }
        catch (const x3::expectation_failure<std::string::const_iterator>& exception)
        {
                std::cout << exception.what() << " " << exception.which() << " " << std::string(exception.where(), text.end()) << std::endl;
        }
}

Error is always: fatal error: template instantiation depth exceeds maximum of 2048

Yes, 2048!!!  Totally stuck

Mike...
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

sehe
On 31-01-16 20:33, Mike Gresens wrote:
Was just the original code from

https://stackoverflow.com/questions/33624149/boost-spirit-x3-cannot-compile-repeat-directive-with-variable-factor/33627991#33627991

replacing std::vector<int> by std::string and x3::int_ by x3::char_

Would like to provide you buildable code, but currently this code seems to
be completely broken:

#include <boost/spirit/home/x3.hpp>
#include <boost/spirit/home/x3/binary.hpp>
#include <string>
#include <memory>
#include <iostream>

namespace x3 = boost::spirit::x3;

namespace hessian {

typedef std::string string_t;

namespace parser {
namespace detail {

//string ::= s b1 b0 <utf8-data> string
//       ::= S b1 b0 <utf8-data>
//       ::= [x00-x1f] <utf8-data>

// NOTE: The length means number of UTF16 characters but the content is
given in UTF8 characters!

struct length_tag;
const auto length_action = [](auto& ctx)
{
	*x3::get<length_tag>(ctx) = x3::_attr(ctx);
};
const auto more_action = [](auto &ctx)
{
	x3::_pass(ctx) = *x3::get<length_tag>(ctx) >  x3::_val(ctx).size();
};
const auto done_action = [](auto &ctx)
{
	x3::_pass(ctx) = *x3::get<length_tag>(ctx) == x3::_val(ctx).size();
};
const auto push_action = [](auto &ctx)
{
	x3::_val(ctx).push_back(x3::_attr(ctx));
};

const x3::rule<class string_rule, string_t> string_rule("string");
const x3::rule<class string1_rule, string_t> string1_rule;
const x3::rule<class string2_rule, string_t> string2_rule;

const auto length_rule = x3::omit[x3::big_word [length_action] ];
const auto content_rule = x3::lexeme[ *(x3::eps [more_action] >> x3::char_
[push_action]) >> x3::eps [done_action] ];

const auto string_rule_def =
x3::with<length_tag>(std::make_shared<std::size_t>())[string1_rule |
string2_rule];
const auto string1_rule_def = x3::lit('S') >> length_rule >> content_rule;
const auto string2_rule_def = x3::lit('s') >> length_rule >> content_rule >>
string_rule;

BOOST_SPIRIT_DEFINE(string_rule, string1_rule, string2_rule);

}

using detail::string_rule;

}
}

void test_string()
{
	const std::string text("s\x00\x07hello, S\x00\x05world"s);
	try
	{
	hessian::string_t attr;
	std::cout << x3::parse(text.begin(), text.end(), x3::eps >
hessian::parser::string_rule, attr) << std::endl;
	std::cout << attr << std::endl;
	}
	catch (const x3::expectation_failure<std::string::const_iterator>&
exception)
	{
		std::cout << exception.what() << " " << exception.which() << " " <<
std::string(exception.where(), text.end()) << std::endl;
	}
}

Error is always: fatal error: template instantiation depth exceeds maximum
of 2048

Here's my take on things:

With debug: http://coliru.stacked-crooked.com/a/40b87b1c08e92806
Without debug: http://coliru.stacked-crooked.com/a/420ba52e76dfeeff

That said, I think you're far better off writing a custom parser here.

#include <iostream>
#include <boost/spirit/home/x3.hpp>
#include <boost/spirit/home/x3/binary.hpp>
#include <string>
#include <memory>

namespace x3 = boost::spirit::x3;

namespace hessian {

typedef std::string string_t;

namespace parser {
namespace detail {

    // string ::= s b1 b0 <utf8-data> string
    //       ::= S b1 b0 <utf8-data>
    //       ::= [x00-x1f] <utf8-data>

    // NOTE: The length means number of UTF16 characters but the content is given in UTF8 characters!

    struct length_tag;
    struct state { size_t expected, actual; };

    const auto length_action = [](auto &ctx) {
        state& s = x3::get<length_tag>(ctx);
        s = { x3::_attr(ctx), 0 };
    };
    const auto more_action = [](auto &ctx) {
        state& s = x3::get<length_tag>(ctx);
        x3::_pass(ctx) = (s.expected >  s.actual);
    };
    const auto done_action = [](auto &ctx) {
        state& s = x3::get<length_tag>(ctx);
        x3::_pass(ctx) = (s.expected == s.actual);
    };
    const auto push_action = [](auto &ctx) { // FIXME handle UTF8
        state& s = x3::get<length_tag>(ctx);
        x3::_val(ctx).push_back(x3::_attr(ctx));
        s.actual += 1;
    };

    const auto length  = x3::omit[x3::big_word[length_action]];
    const auto content = x3::lexeme[*(x3::eps[more_action] >> x3::char_[push_action]) >> x3::eps[done_action]];

    template <typename T> struct mutable_wrapper {
        T mutable value;
        operator T&() const { return value; }
    };

    const auto string_s = 's' >> length >> content;
    const auto string_S = 'S' >> x3::eps >> length >> content;
    const auto string
        = x3::rule<struct string_class, std::string> { "string" }
        = x3::with<length_tag>(mutable_wrapper<state>{})[*string_s >> string_S];
}

auto const& rule = detail::string;
}
}

#include <iomanip>

std::string as_hex(std::string const& what) {
    std::ostringstream oss;
    oss << std::hex << std::setfill('0');
    for (uint8_t ch : what) {
        if (std::isprint(ch)) oss << ch;
        else                  oss << "\\x" << std::setw(2) << int{ch};
    }

    return oss.str();
}

void test_string() {
    const std::string text("s\x00\x07hello, S\x00\x05world", 18);
    try {
        hessian::string_t attr;
        auto f = text.begin(), l = text.end();
        bool ok = x3::parse(f, l, x3::eps > hessian::parser::rule, attr);

        if (ok)
            std::cout << "Parsed: '" << as_hex(attr) << "'\n";
        else
            std::cout << "Failed\n";

        if(f!=l)
            std::cout << "Remaining: '" << as_hex({f,l}) << "'\n";

    } catch (const x3::expectation_failure<std::string::const_iterator> &exception) {
        std::cout << exception.what() << " " << exception.which() << " " << std::string(exception.where(), text.end())
                  << std::endl;
    }
}

int main()
{
    test_string();
}



------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

sehe
On 01-02-16 08:50, Seth wrote:

That said, I think you're far better off writing a custom parser here.

Just for fun, added that too:

http://coliru.stacked-crooked.com/a/87053f2f3f00862d

#include <iostream>
#include <boost/spirit/home/x3.hpp>
#include <boost/fusion/adapted/std_tuple.hpp>
#include <boost/spirit/home/x3/binary.hpp>

namespace x3 = boost::spirit::x3;

namespace hessian {

    typedef std::string string_t;

    namespace parser {

        struct bstring : x3::parser<bstring> {
            using attribute_type = hessian::string_t;

            // string ::= s b1 b0 <utf8-data> string
            //       ::= S b1 b0 <utf8-data>
            //       ::= [x00-x1f] <utf8-data>
            // NOTE: The length means number of UTF16 characters but the content is given in UTF8 characters!
            template <typename It, typename Ctx, typename Attr>
                bool parse(It& f, It const& l, Ctx&, x3::unused_type, Attr& attr) const {
                    auto saved = f;
                    char type;
                    size_t len;
                    auto tied = std::tie(type, len);

                    while (x3::parse(f,l,x3::char_("sS") >> x3::big_word,tied)) {
                        if (!x3::parse(f,l,x3::repeat(len)[x3::char_],attr))
                            break;
                        if (type == 'S')
                            return true;
                    }

                    f = saved;
                    return false;
                }
        };

    }
}

#include <iomanip>

std::string as_hex(std::string const& what) {
    std::ostringstream oss;
    oss << std::hex << std::setfill('0');
    for (uint8_t ch : what) {
        if (std::isprint(ch)) oss << ch;
        else                  oss << "\\x" << std::setw(2) << int{ch};
    }

    return oss.str();
}

void test_string() {
    const std::string text("s\x00\x07hello, S\x00\x05world", 18);
    try {
        hessian::string_t attr;
        auto f = text.begin(), l = text.end();
        bool ok = x3::parse(f, l, x3::eps > hessian::parser::bstring{}, attr);

        if (ok)
            std::cout << "Parsed: '" << as_hex(attr) << "'\n";
        else
            std::cout << "Failed\n";

        if(f!=l)
            std::cout << "Remaining: '" << as_hex({f,l}) << "'\n";

    } catch (const x3::expectation_failure<std::string::const_iterator> &exception) {
        std::cout << exception.what() << " " << exception.which() << " " << std::string(exception.where(), text.end())
                  << std::endl;
    }
}

int main()
{
    test_string();
}


------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

Mike Gresens
Hi sehe,

thanks again! Like always a brilliant job :-)
But I still have no idea, why the recursive grammar breaks X3.

I think I will work with the custom parser approach.
I think there is a good place for UTF8/UTF16 magic too.

Mike...
Reply | Threaded
Open this post in threaded view
|

Re: X3: Using dynamic Repetition Parser Directives

sehe
On 02-02-16 19:27, Mike Gresens wrote:
> But I still have no idea, why the recursive grammar breaks X3.
Haven't reproduced the issue. I wager it would be due to aggressive
inlining. A separation (just an extra `rule<>` or - the big gun - an
`any_parser` would break the cycle?). If you can reproduce it on a
particular compiler with a particular snippet, consider posting it

Seth

------------------------------------------------------------------------------
Site24x7 APM Insight: Get Deep Visibility into Application Performance
APM + Mobile APM + RUM: Monitor 3 App instances at just $35/Month
Monitor end-to-end web transactions and take corrective actions now
Troubleshoot faster and improve end-user experience. Signup Now!
http://pubads.g.doubleclick.net/gampad/clk?id=267308311&iu=/4140
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general