Not able to use x3::skip(skipper)[some recursive rule] in its rule definition

[Han]

I've posted the same question in stackoverflow.

https://stackoverflow.com/questions/45282293/spirit-not-able-to-use-x3skipskippersome-recursive-rule-in-its-rule-defin

Let's say we want to parse a recursive block like this. When "skip_comments_tag" is prefixed to the block, we skip all comments(/*...*/) within this block recursively.

{
    {}
    {
        skip_comments_tag{
            {} /*comments*/ 
            { /*comments*/ } 
        }
    }
}

It's easy to come up with a recursive parser as in Coliru.

namespace Parser {
    auto const ruleComment = x3::lit("/*") >> *(x3::char_ - "*/") >> "*/" | x3::space;

    x3::rule<struct SBlockId> const ruleBlock;
    auto const ruleBlock_def = x3::lit('{') >> *(ruleBlock | "skip_comments_tag" >> x3::skip(ruleComment)[ruleBlock]) >> '}';

    BOOST_SPIRIT_DEFINE(ruleBlock)
}

But it doesn't compile (when the parse function is called) because it will generate an infinite context (by x3::make_context in x3::skip_directive). x3::no_case and x3::with also have this problem because they all use x3::make_context in the implementation.

Questions:

1. Is there always a better way to write parsers for this kind of questions to avoid such compile error and how?
2. Or is the x3::make_context implementation considered to be flawed for this kind of questions?

BTW: TU seperation doesn't solve this. 

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

[cppljevans]

On 07/25/2017 03:40 AM, Han Wang wrote:

> I've posted the same question in stackoverflow.
>
> https://stackoverflow.com/questions/45282293/spirit-not-able-to-use-x3skipskippersome-recursive-rule-in-its-rule-defin
>
> Let's say we want to parse a recursive block like this. When
> "|skip_comments_tag|" is prefixed to the block, we skip all
> comments(|/*...*/|) within this block recursively.
>
> |{{}{skip_comments_tag{{}/*comments*/{/*comments*/}}}}|
>
> It's easy to come up with a recursive parser as in Coliru
> <http://coliru.stacked-crooked.com/a/31fb17064091fae3>.
>
> |namespaceParser{autoconstruleComment =x3::lit("/*")>>*(x3::char_
> -"*/")>>"*/"|x3::space;x3::rule<structSBlockId>construleBlock;autoconstruleBlock_def
> =x3::lit('{')>>*(ruleBlock
> |"skip_comments_tag">>x3::skip(ruleComment)[ruleBlock])>>'}';BOOST_SPIRIT_DEFINE(ruleBlock)}|
>
> But it doesn't compile (when the |parse| function is called) because it
> will generate an infinite context (by |x3::make_context| in
> |x3::skip_directive|). |x3::no_case| and |x3::with| also have this
> problem because they all use |x3::make_context| in the implementation.
>
> Questions:
>
>  1. Is there always a better way to write parsers for this kind of
>     questions to avoid such compile error and how?
>  2. Or is the |x3::make_context| implementation considered to be flawed
>     for this kind of questions?
>
> BTW: TU seperation doesn't solve this.
>

Hi Han.

You might try the get_rhs branch of a spirit fork here:

https://github.com/cppljevans/spirit/tree/get_rhs

and then #define BOOST_SPIRIT_GET_RHS_CRTP.  As you can see from here:

https://github.com/cppljevans/spirit/blob/get_rhs/include/boost/spirit/home/x3/nonterminal/detail/rule.hpp#L140

that will disable the x3::make_context calls.

Please let me know if that works.  If so, I might be
encouraged enough to go to the trouble of creating a pull request
despite the need (as pointed out in a private email) that
I'll need to document the rationale for the pull request.

HTH.

-regards,
Larry



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Han]

Hi Larry, 

Thanks. I've tried this get_rhs branch but it doesn't solve this problem. Still get the same error. 

Best regards,

Han

On Tue, Jul 25, 2017 at 1:21 PM, Larry Evans <[hidden email]> wrote:

On 07/25/2017 03:40 AM, Han Wang wrote:

I've posted the same question in stackoverflow.

https://stackoverflow.com/questions/45282293/spirit-not-able-to-use-x3skipskippersome-recursive-rule-in-its-rule-defin

Let's say we want to parse a recursive block like this. When "|skip_comments_tag|" is prefixed to the block, we skip all comments(|/*...*/|) within this block recursively.

|{{}{skip_comments_tag{{}/*comments*/{/*comments*/}}}}|

It's easy to come up with a recursive parser as in Coliru <http://coliru.stacked-crooked.com/a/31fb17064091fae3>.

|namespaceParser{autoconstruleComment =x3::lit("/*")>>*(x3::char_ -"*/")>>"*/"|x3::space;x3::rule<structSBlockId>construleBlock;autoconstruleBlock_def =x3::lit('{')>>*(ruleBlock |"skip_comments_tag">>x3::skip(ruleComment)[ruleBlock])>>'}';BOOST_SPIRIT_DEFINE(ruleBlock)}|

But it doesn't compile (when the |parse| function is called) because it will generate an infinite context (by |x3::make_context| in |x3::skip_directive|). |x3::no_case| and |x3::with| also have this problem because they all use |x3::make_context| in the implementation.

Questions:

 1. Is there always a better way to write parsers for this kind of
    questions to avoid such compile error and how?
 2. Or is the |x3::make_context| implementation considered to be flawed
    for this kind of questions?

BTW: TU seperation doesn't solve this.

Hi Han.

You might try the get_rhs branch of a spirit fork here:

https://github.com/cppljevans/spirit/tree/get_rhs

and then #define BOOST_SPIRIT_GET_RHS_CRTP.  As you can see from here:

https://github.com/cppljevans/spirit/blob/get_rhs/include/boost/spirit/home/x3/nonterminal/detail/rule.hpp#L140

that will disable the x3::make_context calls.

Please let me know if that works.  If so, I might be
encouraged enough to go to the trouble of creating a pull request
despite the need (as pointed out in a private email) that
I'll need to document the rationale for the pull request.

HTH.

-regards,
Larry



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

[cppljevans]

On 07/25/2017 07:43 AM, Han Wang wrote:
> Hi Larry,
>
> Thanks. I've tried this get_rhs branch but it doesn't solve this
> problem. Still get the same error.
>
> Best regards,
> Han
>
[snip]

OPPS.  Sorry about that.  I should have read where you say in OP:

it will generate an infinite context (by x3::make_context in
x3::skip_directive).

I just saw the x3::make_context and remembered that was causing
long compile times.  There's a make_unique_context which
avoids the infinite recursion.  When that's applied, as shown here:

https://github.com/cppljevans/spirit/blob/get_rhs/include/boost/spirit/home/x3/directive/skip.hpp#L40

Compile succeeds fast:

make compile
time
/home/evansl/dwnlds/llvm/3.9/prebuilt/clang+llvm-3.9.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++
-c -O0 -stdlib=libc++  -std=c++1z -ftemplate-backtrace-limit=0
-fdiagnostics-show-template-tree -fno-elide-type
-fmacro-backtrace-limit=0     -c -std=c++14 -ftemplate-depth=1024
-I/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include
-I/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0
-DBOOST_SPIRIT_GET_RHS_CRTP=0 -DBOOST_SPIRIT_ATTR_XFORM_IN_RULE=0  main.cpp
In file included from main.cpp:7:
In file included from
/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include/boost/spirit/home/x3.hpp:17:
In file included from
/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include/boost/spirit/home/x3/nonterminal.hpp:10:
/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include/boost/spirit/home/x3/nonterminal/rule.hpp:18:11:
warning:
       deprecated. May cause link error when using
BOOST_SPIRIT_INSTANTIATE. [-W#pragma-messages]
   #pragma message "deprecated.  May cause link error when using
BOOST_SPIRIT_INSTANTIATE."
           ^
/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include/boost/spirit/home/x3/nonterminal/rule.hpp:22:11:
warning:
       deprecated. May cause excessive compile times when many rules.
[-W#pragma-messages]
   #pragma message "deprecated.  May cause excessive compile times when
many rules."
           ^
2 warnings generated.

real 0m2.351s
user 0m1.372s
sys 0m0.072s

Compilation finished at Tue Jul 25 08:53:02

HTH.

-regards,
Larry




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

[Han]

Hi Larry, 

I think we cannot just change make_context to make_unique_context here in x3::skip_directive. If there already is a skipper in the context (x3::space e.g), the new skipper (ruleComment) will be ignored (because x3::make_unique_context will simply return the original context). 

Best regards,

Han

On Tue, Jul 25, 2017 at 4:02 PM, Larry Evans <[hidden email]> wrote:

On 07/25/2017 07:43 AM, Han Wang wrote:

Hi Larry,

Thanks. I've tried this get_rhs branch but it doesn't solve this problem. Still get the same error.

Best regards,
Han

[snip]

OPPS.  Sorry about that.  I should have read where you say in OP:

it will generate an infinite context (by x3::make_context in x3::skip_directive).

I just saw the x3::make_context and remembered that was causing
long compile times.  There's a make_unique_context which
avoids the infinite recursion.  When that's applied, as shown here:

https://github.com/cppljevans/spirit/blob/get_rhs/include/boost/spirit/home/x3/directive/skip.hpp#L40

Compile succeeds fast:

make compile
time /home/evansl/dwnlds/llvm/3.9/prebuilt/clang+llvm-3.9.0-x86_64-linux-gnu-ubuntu-16.04/bin/clang++ -c -O0 -stdlib=libc++  -std=c++1z -ftemplate-backtrace-limit=0 -fdiagnostics-show-template-tree -fno-elide-type -fmacro-backtrace-limit=0     -c -std=c++14 -ftemplate-depth=1024 -I/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include -I/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0 -DBOOST_SPIRIT_GET_RHS_CRTP=0 -DBOOST_SPIRIT_ATTR_XFORM_IN_RULE=0  main.cpp
In file included from main.cpp:7:
In file included from /home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include/boost/spirit/home/x3.hpp:17:
In file included from /home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include/boost/spirit/home/x3/nonterminal.hpp:10:
/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include/boost/spirit/home/x3/nonterminal/rule.hpp:18:11: warning:
      deprecated. May cause link error when using BOOST_SPIRIT_INSTANTIATE. [-W#pragma-messages]
  #pragma message "deprecated.  May cause link error when using BOOST_SPIRIT_INSTANTIATE."
          ^
/home/evansl/prog_dev/boost/releases/ro/boost_1_64_0/sandbox/pull_requests/context_debug_fix.fork/spirit/include/boost/spirit/home/x3/nonterminal/rule.hpp:22:11: warning:
      deprecated. May cause excessive compile times when many rules. [-W#pragma-messages]
  #pragma message "deprecated.  May cause excessive compile times when many rules."
          ^
2 warnings generated.

real    0m2.351s
user    0m1.372s
sys     0m0.072s

Compilation finished at Tue Jul 25 08:53:02

HTH.

-regards,
Larry




------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

[cppljevans]

On 07/25/2017 09:18 AM, Han Wang wrote:
> Hi Larry,
>
> I think we cannot just change make_context to make_unique_context here
> in x3::skip_directive. If there already is a skipper in the context
> (x3::space e.g), the new skipper (ruleComment) will be ignored (because
> x3::make_unique_context will simply return the original context).
>
[snip]
Good catch.  I had assumed that context<ID,val,NEXT> was a dictionary
with only 1 value for given ID.  However, I guess there could be
several entries where ID==skipper_tag and only the 1st one is retrieved
by the context.get<ID_>(ID_) member function.  I guess there's some
method to pop the top value with given ID off the context stack.

In that case, I've no idea how to solve the problem.

Sorry :(

-regards,
Larry


------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

[cppljevans]

On 07/25/2017 09:54 AM, Larry Evans wrote:

> On 07/25/2017 09:18 AM, Han Wang wrote:
>> Hi Larry,
>>
>> I think we cannot just change make_context to make_unique_context here
>> in x3::skip_directive. If there already is a skipper in the context
>> (x3::space e.g), the new skipper (ruleComment) will be ignored
>> (because x3::make_unique_context will simply return the original
>> context).
>>
> [snip]
> Good catch.  I had assumed that context<ID,val,NEXT> was a dictionary
> with only 1 value for given ID.  However, I guess there could be
> several entries where ID==skipper_tag and only the 1st one is retrieved
> by the context.get<ID_>(ID_) member function.  I guess there's some
> method to pop the top value with given ID off the context stack.
>
> In that case, I've no idea how to solve the problem.
>
> Sorry :(
>
> -regards,
> Larry
>

I kept trying and *maybe* this:

https://github.com/cppljevans/spirit/blob/get_rhs/include/boost/spirit/home/x3/directive/skip.hpp#L85

will solve problem.  It did successfully compile & run your test
program.

HTH.

-regards,
Larry



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

[cppljevans]

On 07/26/2017 08:37 AM, Larry Evans wrote:

> On 07/25/2017 09:54 AM, Larry Evans wrote:
>> On 07/25/2017 09:18 AM, Han Wang wrote:
>>> Hi Larry,
>>>
>>> I think we cannot just change make_context to make_unique_context
>>> here in x3::skip_directive. If there already is a skipper in the
>>> context (x3::space e.g), the new skipper (ruleComment) will be
>>> ignored (because x3::make_unique_context will simply return the
>>> original context).
>>>
>> [snip]
>> Good catch.  I had assumed that context<ID,val,NEXT> was a dictionary
>> with only 1 value for given ID.  However, I guess there could be
>> several entries where ID==skipper_tag and only the 1st one is retrieved
>> by the context.get<ID_>(ID_) member function.  I guess there's some
>> method to pop the top value with given ID off the context stack.
>>
>> In that case, I've no idea how to solve the problem.
>>
>> Sorry :(
>>
>> -regards,
>> Larry
>>
>
> I kept trying and *maybe* this:
>
> https://github.com/cppljevans/spirit/blob/get_rhs/include/boost/spirit/home/x3/directive/skip.hpp#L85 
>
>
> will solve problem.  It did successfully compile & run your test
> program.
>
> HTH.
>

Further revision were made to remove the code in skip.hpp which was,
essentially, close duplication of code in context.hpp.  This was done
by just modifying the context.hpp detail::make_unique_context slightly.

HTH.

-regards,
Larry



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

[cppljevans]

On 07/27/2017 02:48 AM, Larry Evans wrote:

> On 07/26/2017 08:37 AM, Larry Evans wrote:
>> On 07/25/2017 09:54 AM, Larry Evans wrote:
>>> On 07/25/2017 09:18 AM, Han Wang wrote:
>>>> Hi Larry,
>>>>
>>>> I think we cannot just change make_context to make_unique_context
>>>> here in x3::skip_directive. If there already is a skipper in the
>>>> context (x3::space e.g), the new skipper (ruleComment) will be
>>>> ignored (because x3::make_unique_context will simply return the
>>>> original context).
>>>>
>>> [snip]
>>> Good catch.  I had assumed that context<ID,val,NEXT> was a dictionary
>>> with only 1 value for given ID.  However, I guess there could be
>>> several entries where ID==skipper_tag and only the 1st one is retrieved
>>> by the context.get<ID_>(ID_) member function.  I guess there's some
>>> method to pop the top value with given ID off the context stack.
>>>
>>> In that case, I've no idea how to solve the problem.
>>>
>>> Sorry :(
>>>
>>> -regards,
>>> Larry
>>>
>>
>> I kept trying and *maybe* this:
>>
>> https://github.com/cppljevans/spirit/blob/get_rhs/include/boost/spirit/home/x3/directive/skip.hpp#L85 
>>
>>
>> will solve problem.  It did successfully compile & run your test
>> program.
>>
>> HTH.
>>
> Further revision were made to remove the code in skip.hpp which was,
> essentially, close duplication of code in context.hpp.  This was done
> by just modifying the context.hpp detail::make_unique_context slightly.
>

[snip]
After further thought, the proposed solution may not always work.
I think it *may* suffer the same problem as before in some cases.
One such case maybe where the:

    make_unique_context<skipper_tag>(skipper,context)

is called with alternate values for skipper, say, skipper1, skipper2.
Then the context would be:

   context<skipper_tag>
   ( skipper1
   , context<skipper_tag>
     ( skipper2
     , context<skipper_tag>
       ( skipper1
       , unused_type
       )
     )
   )

I've no idea what type of grammar might produce this; however, before
this thread started, I had no idea would type of grammar would produce
your particular problem, which, simplified, produces:

   context<skipper_tag>
   ( skipper1
   , context<skipper_tag>
     ( skipper1
     , context<skipper_tag>
       ( skipper1
       , unused_type
       )
     )
   )

IOW, each time the skip_directive::parse is called, the context is
extended.  If the skip_directive is in a recursive rule, then it
get's extended without end.

I've no idea, yet, how to solve this problem.  Also, I've not
made a test case either so I may be completely wrong.

I hope someone else will solve problem.

-regards,
Larry



------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general