What do you intuitively expect that the action of the skipper should be in the code that you posted? That it only trims spaces from the extremes of the input string, and doesn't do anything else?
It could work that way but I think it would be a lot less useful.
In qi, skippers apply to rules, and get activated once before each expression in the rule.
You can have different skippers for different rules. In your example though, there is only one rule. I think it's pretty reasonable to expect that if you apply `qi::blank` to the rule `*(qi::char_("a-z"))` that the results will be the same as `*(qi::char_("a-z"))` acting on a string where all the whitespace was deleted beforehand.
In pure qi there is not a notion of tokens. If you are trying to write a (typical) tokenizer using qi, you probably don't want to use `qi::blank` as a skipper for that. Or you could, but then you should use the "lexeme" directive for the part of the grammar which matches tokens, as that will prevent this "token merging" issue you raise.
> Why does +qi::char_("a-z") to fully match "ab cd" with qi::blank skipper?
Because you are using phrase_parse. Most parsers in Spirit start by
invoking the skipper. The char parser is no different.
Since you are wrapping the char parser in +, it will be called
repeatedly, and for each invokation, it first calls the skipper:
if (first != last && this->derived().test(*first, context))
> Why do the tokens "ab" and "cd" get merged?
They are not treated like separate tokens at all. It just happens that
the space is being consumed by the char parser calling to your skipper
before it actually sees 'c'.
> I just spent 2h+ chasing this.
I consider skippers a different approach to lexing.
However, I don't have any active experience with them, since the project
I am working on is whitespace sensitive, therefore I opted to always
make spacing explicit in my grammar and not use phrase_parse.
Yes I understood the way it works, after being bitten by it.
My post was just a bit of a rant :-) after spending the whole day reading
spirit documentation and still not getting my simple parser to work.
On 10/30/15 2:53 AM, Sanchay Harneja wrote:
> Yes I understood the way it works, after being bitten by it.
> My post was just a bit of a rant :-) after spending the whole day reading
> spirit documentation and still not getting my simple parser to work.
In such cases, I rely on #define BOOST_SPIRIT_QI_DEBUG to help me
visualize what's going on.