Rethinking the "what"

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Rethinking the "what"

Joel de Guzman-2
Hi,

I have mixed feelings with the way we have the "what"
member functions. In spirit2, it simply returns a
std::string. I've had problems when I had to return
a wide string. No good. In spirit2x, I initially redesigned
it to take in a templated stream:. Here's the int parser
parser's what function, for example:

     template <typename OStream, typename Context>
     void what(OStream& out, Context const& /*context*/) const
     {
         out << "integer";
     }

Here's for alternative:

     template <typename OStream, typename Context>
     void what(OStream& out, Context const& context) const
     {
         out << "alternatives[";
         fusion::for_each(elements,
             spirit::detail::what_function<OStream, Context>(out, context));
         out << "]";
     }

Iterates over the alternatives and collects the "what"s of each.

So far so good. But then I had to implement the expect parser
which throws an exception. Somewhere in its internals, it
has to call the "what" function of the parser that fails
and make an exception out of it. In spirit2 (with the what function
returning string) the code was:

     Exception x = {first, last, component.what(context) };
     throw x;

Now, with spirit2x, I had to create a string stream. Yaiks!

1) We all know that this is an expensive operation
2) What string type? Base it on the iterator type?
    What if the iterator type points to a stream of tokens?

Also, I'm not quite happy that the what function returns
an unstructured blob. Say, we have this parser:

     start = '[' > (a | b | c);

The default error message extracted from this when > fails is
something like:

     "Error. Expecting alternatives[a, b, c] here"

Gibberish, to say the least. We'd want to transform that to something
more meaningful. Alas, the unformatted stream (or string) makes it
difficult. So what do we want? I'm thinking we want to return an
AST of the parser! Something that can be travesed.

So, I came up with something python/lua/-ish dynamic type info:

     struct info
     {
         struct nil {};
         struct nary;

         typedef
             boost::variant<
                 nil
               , int
               , double
               , std::basic_string<char>
               , std::basic_string<wchar_t>
               , boost::recursive_wrapper<nary>
             >
         type;

         info(std::string const& tag)
           : tag(tag), value(nil()) {}

         template <typename T>
         info(std::string const& tag, T const& value)
           : tag(tag), value(value) {}

         std::string tag;
         type value;
     };

     struct info::nary
     {
         template <typename Iterator>
         nary(Iterator first, Iterator last)
           : children(first, last) {}

         std::vector<info> children;
     };

with this structure, we can easily transform the error message
into something like:

     "Error. Expecting a or b or c"

we can even use karma to walk the tree and therefore provide more
powerful output transformations.

Your thoughts?

Regards,
--
Joel de Guzman
http://www.boostpro.com
http://spirit.sf.net


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Spirit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-devel
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking the "what"

Joel de Guzman-2
Joel de Guzman wrote:

> The default error message extracted from this when > fails is
> something like:
>
>      "Error. Expecting alternatives[a, b, c] here"
>
> Gibberish, to say the least.

Bad example. That wasn't gibberish, it seems. But I think you
get my point :P .. Just imagine a bit more complex expression.

Regards,
--
Joel de Guzman
http://www.boostpro.com
http://spirit.sf.net


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Spirit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-devel
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking the "what"

CARL BARRON-3
In reply to this post by Joel de Guzman-2

On Nov 21, 2008, at 11:45 PM, Joel de Guzman wrote:


So, I came up with something python/lua/-ish dynamic type info:

    struct info
    {
        struct nil {};
        struct nary;

        typedef
            boost::variant<
                nil
              , int
              , double
              , std::basic_string<char>
              , std::basic_string<wchar_t>
              , boost::recursive_wrapper<nary>

        type;

        info(std::string const& tag)
          : tag(tag), value(nil()) {}

        template <typename T>
        info(std::string const& tag, T const& value)
          : tag(tag), value(value) {}

        std::string tag;
        type value;
    };

    struct info::nary
    {
        template <typename Iterator>
        nary(Iterator first, Iterator last)
          : children(first, last) {}

        std::vector<info> children;
    };

with this structure, we can easily transform the error message
into something like:

    "Error. Expecting a or b or c"

we can even use karma to walk the tree and therefore provide more
powerful output transformations.

Your thoughts?

 You can do almost anything with this.  It is a pain to re-parse the given 'what is wrong' stream .   Karma or other tree walker
can walk the tree.

 Now for clarification suppose

r =   a > b > c;
b =  d > e;
on_error(r,handler());


now e fails to parse what will this 'what parameter' produce if e fails?
I assume it will point the error after the pre-skip [if any] of e.




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Spirit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-devel
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking the "what"

Joel de Guzman-2
Carl Barron wrote:

>  Now for clarification suppose
>
> r =   a > b > c;
> b =  d > e;
> on_error(r,handler());
>
>
> now e fails to parse what will this 'what parameter' produce if e fails?
> I assume it will point the error after the pre-skip [if any] of e.

Exactly. Well, almost... Regarding pre-skipping, e will pre-skip
only if it is a terminal. Perhaps it's a good idea to always do
a pre-skip before throwing the exception. That way, the location
of the error is dead accurate.

Regards,
--
Joel de Guzman
http://www.boostpro.com
http://spirit.sf.net


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Spirit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-devel
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking the "what"

Hartmut Kaiser
In reply to this post by Joel de Guzman-2
> I have mixed feelings with the way we have the "what"
> member functions. In spirit2, it simply returns a
> std::string. I've had problems when I had to return
> a wide string. No good. In spirit2x, I initially redesigned
> it to take in a templated stream:. Here's the int parser
> parser's what function, for example:
>
>      template <typename OStream, typename Context>
>      void what(OStream& out, Context const& /*context*/) const
>      {
>          out << "integer";
>      }
>
> Here's for alternative:
>
>      template <typename OStream, typename Context>
>      void what(OStream& out, Context const& context) const
>      {
>          out << "alternatives[";
>          fusion::for_each(elements,
>              spirit::detail::what_function<OStream, Context>(out,
> context));
>          out << "]";
>      }
>
> Iterates over the alternatives and collects the "what"s of each.
>
> So far so good. But then I had to implement the expect parser
> which throws an exception. Somewhere in its internals, it
> has to call the "what" function of the parser that fails
> and make an exception out of it. In spirit2 (with the what function
> returning string) the code was:
>
>      Exception x = {first, last, component.what(context) };
>      throw x;
>
> Now, with spirit2x, I had to create a string stream. Yaiks!
>
> 1) We all know that this is an expensive operation
> 2) What string type? Base it on the iterator type?
>     What if the iterator type points to a stream of tokens?
>
> Also, I'm not quite happy that the what function returns
> an unstructured blob. Say, we have this parser:
>
>      start = '[' > (a | b | c);
>
> The default error message extracted from this when > fails is
> something like:
>
>      "Error. Expecting alternatives[a, b, c] here"
>
> Gibberish, to say the least. We'd want to transform that to something
> more meaningful. Alas, the unformatted stream (or string) makes it
> difficult. So what do we want? I'm thinking we want to return an
> AST of the parser! Something that can be travesed.
>
> So, I came up with something python/lua/-ish dynamic type info:
>
>      struct info
>      {
>          struct nil {};
>          struct nary;
>
>          typedef
>              boost::variant<
>                  nil
>                , int
>                , double
>                , std::basic_string<char>
>                , std::basic_string<wchar_t>
>                , boost::recursive_wrapper<nary>
>              >
>          type;
>
>          info(std::string const& tag)
>            : tag(tag), value(nil()) {}
>
>          template <typename T>
>          info(std::string const& tag, T const& value)
>            : tag(tag), value(value) {}
>
>          std::string tag;
>          type value;
>      };
>
>      struct info::nary
>      {
>          template <typename Iterator>
>          nary(Iterator first, Iterator last)
>            : children(first, last) {}
>
>          std::vector<info> children;
>      };
>
> with this structure, we can easily transform the error message
> into something like:
>
>      "Error. Expecting a or b or c"
>
> we can even use karma to walk the tree and therefore provide more
> powerful output transformations.

A couple of (possibly unrelated) thought:

I'ld say, make it as simple as possible and let the user supply the unknowns (like the type of the stream to use in your example above).

Letting what() to create a parse tree duplicates the parse tree the user might want to create in the first place. And it adds complexity even if the user doesn't need it. The scheme we have had in Spirit2 was not ideal, but sufficient. What's perhaps missing there is the possibility for the user to customize the behavior.

Creating expensive object instances during error handling is generally not an issue, at least that's the way I'm approaching error handling...

Don't add more complexity to the parsing, compilation times are already scary without this.

My 2€ cents.
Regards Hartmut




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Spirit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-devel
Reply | Threaded
Open this post in threaded view
|

Re: Rethinking the "what"

Joel de Guzman-2
Hartmut Kaiser wrote:

> A couple of (possibly unrelated) thought:
>
> I'ld say, make it as simple as possible and let the user supply the unknowns
> (like the type of the stream to use in your example above).

That's what I did with Spirit2x. It works ok, until I had to write
the expect parser where we have to supply a stream. Recall that
the expectation_failure exception requires the parser to know
"what" it is. There is no way in our interface to make that user
configurable, unless we change the expect API into something else,
which I do not want to do.

> Letting what() to create a parse tree duplicates the parse tree the user
> might want to create in the first place.

No, that's a different parse tree. This is a grammar parse tree.
In fact, the what function is emitting a flat version of it.
In terms of complexity, it should be the same.

> And it adds complexity even if the
> user doesn't need it. The scheme we have had in Spirit2 was not ideal, but
> sufficient.

Not sure I agree. It was never sufficient. Beyond some toy examples,
it will be difficult to coax the error result into something more useful
(*** but see below)

> What's perhaps missing there is the possibility for the user to
> customize the behavior.

Sure, that was what I planned in 2x. Alas, it's not doable unless
we make expect a directive, like:

     p >> expect<exception>()[p2]

ala classic Spirit. I don't like that. IMO, the spirit2 expect syntax
is as ideal as it can be. Let's not mess with perfection ;-) I'd
like to tweak something else to cater to the expect syntax, not the
other way around.

> Creating expensive object instances during error handling is generally not an
> issue, at least that's the way I'm approaching error handling...

Yes, that's a lesser issue. That's not my main concern.

> Don't add more complexity to the parsing, compilation times are already scary
> without this.

I'm not adding more complexity to parsing. "what" is not called at parse
time. Besides, the info object I suggested is not a template. It is
instantiated only once. It's as cheap (at compile time) as any plain
OOP class.


(***) Having said all that, a minimalist approach would be to generate
something like:

     struct info
     {
         std::string tag;
         boost::any data;
         std::list<info> children;
     };

tag: parser tag, e.g. "int_"
data: typically a std::string or wide-string
children: non-empty for composites like sequence or alternative

It is also my observation that when you have an expect expression like:

     a > b

you do not want b to be a complex expression. In order to generate
simple error meessages like:

     "Error. Expecting 'begin'" // a literal

or

     "Error. Expecting 'int'" // int_

or

     "Error. Expecting conditional-expression"  // a rule

you'll want b above to be a simple terminal or a rule. So, if tag is for
a moderately complex expression like "alternative", the user can have a
std::map of functions that walks the info.children transforming that
into something like, e.g.:

     // alternates
     "Error. Expecting conditional-expression or integer or 'begin'"

it's probably a good idea to suggest proper usage of the expect
expression to avoid overly complex error messages.

Regards,
--
Joel de Guzman
http://www.boostpro.com
http://spirit.sf.net


-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
Spirit-devel mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-devel