Horrible compiletimes and memory usage while compiling a parser with X3

classic Classic list List threaded Threaded
55 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Horrible compiletimes and memory usage while compiling a parser with X3

Exagon
This post was updated on .
I am currently implementing expressions and the operator hierarchy for my DSL, using boost spirit X3.
I think my parser is semanticaly correct, but when I try to compile it, while compiling, gcc and clang having HUGE memory footprint, compiles for an infinite amount of time, and then quits with "g++: internal compiler error: Killed (program cc1plus)".
I tried to minimize the code arround the expression parser, but its somehow not that trivial.
Here is an example of what causes the compiler error. I dont know if this is a bug, or if I have a semantic bug in my code, or if this is a bug.

I think the problem is somwhere there:

    auto const idx = as<ast::Operation>(helper::idxaccess_op > expression > ']');
    auto const func = as<ast::Operation>(helper::func_call_op > expression%',' > ')');
    auto const data = as<ast::Operation>(helper::access_op > expression);

    auto const func_call_expr_def =
        primary_expr >> *(idx|func|data);

if I change (idx|func|data) to (idx|func), it also compiles forever and uses up to 16GB of ram, but gcc is able to compile it, and the parser works how it should work.

Right now this causes compiletimes up to 1h, which makes the expression parser realy unusable right now

Thanks
Exagon
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Jens Kallup
Hi,

There a 3 problems that can be:

1 - to small memory (a developer PC should be 4 GB of RAM today)
2 - the fact is, that boost is very compilcated template system.
3 - You have a too small swap partition (Linux), and/or user file fs

You can test, if you increase the memory.
Or You can try to use pre-compiled-header-file(s), what can be use
by gnu g++ to generate (very big) header binary, which makes the
compile time a little bit charmed/faster.
BTW: PCH files can use for other Frameworks/Projects, too.

You can try/test to increase the number of open file id's.
Or increase the virtual template depth

Jens

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Exagon
This post was updated on .
I have 32Gb RAM, while compiling it uses up to 24GB of this RAM.
If I add "-ftemplate-depth=4096" in clang, it compiles, but uses all of the RAM, and compiling takes up to an hour. Is this a bug in spirit X3, or will I have to deal with this in the future?
EDIT: I have a 64GB swap partition and 32 GB RAM, I think this should be enough.
The Internal compiler error was because I had "only" 8Gb of ram and 8GB swap on my notebook.
Now on my PC, where I have enough memory, the memory usage and the huge compiletimes making it a pain to work with X3, with my expression parser

EDIT II: i just recognized the resulting binary is 1.6 GB big :/
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Jens Kallup

On 10/23/2016 10:05 PM, Exagon wrote:
> Is this a bug in spirit X3, or will
> I have to deal with this in the future?

No, You definitive #include to many headers in the sources.
Try/Find out what You need, and delete not used.

BTW:

#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>
#include <boost/spirit/include/phoenix.hpp>

should do the magic, because boost is self explained. So,
You don't have to deal with all of the headers.

Jens

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Exagon
have you looked at my example? I dont include anything I dont need, and I dont think including boost/spirit/include/qi.hpp will solve anything, since I use boost spirit x3.
Have a look at the example I made, its minimal, nothing fancy, and compile times up to 1h are definitely not caused by the 3 headers I included in the example.
Greetings Exagon
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Jens Kallup
Oh, sorry, I see.
It seems that you use the spirit X3 version, instead of
boost::spirit
I recommend to use boost version.
I can't say, who maintain it, but I had no concerns.
Please download latest bosst version, compile and install it.
Then you can have a look over the code above the mail text.
It doesn't work for my need, but it can be an example, on how
to code with boost::soirit.
If you have ideas, why the source can't parse dynamic text:

"class ttt of ttt endclass"   but:
"class of endclass"

let me know.

Hope this helps

Jens

#define BOOST_SPIRIT_DEBUG
#define BOOST_SPIRIT_ACTIONS_ALLOW_ATTR_COMPAT

#include <boost/config/warning_disable.hpp>
#include <boost/spirit/include/qi.hpp>

#include <boost/spirit/include/phoenix_core.hpp>
#include <boost/spirit/include/phoenix_operator.hpp>
#include <boost/spirit/include/phoenix_fusion.hpp>
#include <boost/spirit/include/phoenix_stl.hpp>
#include <boost/spirit/include/phoenix_object.hpp>

#include <string>
#include <iostream>

#define ÙSE_QT
#ifdef  USE_QT
#include <QMessageBox>
#endif

using namespace std;
using namespace boost::spirit;

namespace client
{
      namespace fusion = boost::fusion;
      namespace phoenix = boost::phoenix;

      namespace qi = boost::spirit::qi;
      namespace ascii = boost::spirit::ascii;

      template <typename Iterator>
      struct dbase_skipper : public qi::grammar<Iterator>
      {
          dbase_skipper() : dbase_skipper::base_type(my_skip, "dBase")
          {
              using qi::ascii::char_;
              using qi::ascii::space;
              using qi::eol;
              using qi::eoi;

              my_skip = (char_("[ \t\n\r]"))                            |
              ("**" >> *(char_ - eol) >> (eol | eoi | char_("[\n\r]"))) |
              ("&&" >> *(char_ - eol) >> (eol | eoi | char_("[\n\r]"))) |
              ("//" >> *(char_ - eol) >> (eol | eoi | char_("[\n\r]"))) |
              ("/*" >> *(char_ - "*/") >> "*/")
              ;

              BOOST_SPIRIT_DEBUG_NODE((my_skip));
          }
          qi::rule<Iterator> my_skip;
      };

      template <typename Iterator, typename Skipper =
dbase_skipper<Iterator>>
      struct dbase_grammar : public qi::grammar<Iterator, Skipper>
      {
          qi::rule<Iterator, Skipper> start, run_app;
          qi::rule<Iterator, Skipper> block;
          qi::rule<Iterator, Skipper> statement;

          dbase_grammar() : dbase_grammar::base_type(start)
          {
              using qi::lit;
              using qi::char_;

              using qi::on_error;
              using qi::fail;

              using phoenix::construct;
              using phoenix::val;


              start = run_app.alias();

              run_app = - symbol;

              expression =
                      term.alias()
                      >> *(
                        ('+' >> term )
                      | ('-' >> term ))
                      ;
              term =
                      factor.alias()
                      >> *(
                        ('*' >> factor )
                      | ('/' >> factor ))
                      ;

              factor =
                      ( symbol_digit
                      ///| symbol_alpha
                      )
                      >> *(
                      ('('   >> expression >> ')')
                      | ('-' >> factor     )
                      | ('+' >> factor     ))
                      ;

              symbol =
                  (((symbol_class > //symbol_space >
                     //symbol_alpha > //symbol_space >
                     symbol_of    > //symbol_space >
                     //symbol_alpha > //symbol_space >
                     symbol_endclass >> run_app) |
                    (symbol_class > //symbol_space >
                     //symbol_alpha > //symbol_space >
                     symbol_of    > //symbol_space >
                     //symbol_alpha > //symbol_space >
                     symbol_endclass)
                    )              |
                  ((symbol_alpha  > qi::char_('=') > expression >>
run_app) |
                   (symbol_alpha  > qi::char_('=') > expression           ))
                  )
                  ;

              symbol_class    = "class";
              symbol_of       = "of";
              symbol_endclass = "endclass";

              symbol_space =
                  +(qi::char_("[ \t\n\r]") | eol | eoi)
                  ;

              symbol_alpha =
                  +(qi::alpha | qi::digit | qi::char_( "_" ))
                  ;

              symbol_digit =
                  +(qi::digit)
                  ;

              on_error<fail>
              (
                  start
                , std::cout
                      << val("Error! Expecting ")
                      << _4                               // what failed?
                      << val(" here: \"")
                      << construct<std::string>(_3, _2)   // iterators
to error-pos, end
                      << val("\"")
                      << std::endl
              );


              BOOST_SPIRIT_DEBUG_NODE(start);
              BOOST_SPIRIT_DEBUG_NODE(symbol);
              BOOST_SPIRIT_DEBUG_NODE(symbol_of);
              BOOST_SPIRIT_DEBUG_NODE(symbol_endclass);
              BOOST_SPIRIT_DEBUG_NODE(symbol_class);
              BOOST_SPIRIT_DEBUG_NODE(symbol_space);
              BOOST_SPIRIT_DEBUG_NODE(symbol_alpha);
              BOOST_SPIRIT_DEBUG_NODE(symbol_digit);


              BOOST_SPIRIT_DEBUG_NODE(expression);
              BOOST_SPIRIT_DEBUG_NODE(term);
              BOOST_SPIRIT_DEBUG_NODE(factor);
          }

          qi::rule<Iterator, Skipper>
          symbol,
          symbol_alpha,
          symbol_digit,
          symbol_space,
          symbol_class,
          symbol_endclass,
          symbol_of;

          qi::rule<Iterator, Skipper> expression, term, factor;
      };
}

bool my_parser(std::string const str)
{
      typedef std::string::const_iterator iterator_t;

      typedef client::dbase_grammar <iterator_t> grammar;
      typedef client::dbase_skipper <iterator_t> skipper;

      grammar pg;
      skipper skp;

      iterator_t iter = str.begin();
      iterator_t end  = str.end();

      bool r = phrase_parse(iter, end, pg, skp);
      if (r == true) {
          #ifdef USE_QT
          QMessageBox::information(0,"Parser", "Parsing SUCCESS.");
          #else
          std::cout << "SUCCESS" << std::endl;
          #endif
          return true;
      }

      if (iter != end) {
          std::cout << "Remaining: '" << std::string(iter, end) <<
std::endl;
          #ifdef USE_QT
          QMessageBox::information(0,"Parser", "Parsing ERROR.");
          #else
          std::cout << "ERROR" << std::endl;
          #endif
          return false;
      }   return false;
}

bool parseText(std::string str, int mode)
{
      return my_parser(str);
}

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Mikael Asplund
In reply to this post by Exagon
Hi!

Yes, X3 is very nice in many ways, but template depth really gets to be a problem when you create a more complicated grammar.

I have some experience implementing expressions too, but I have to make Visual Studio compile my code, and it doesn't even seem to have an option like "-ftemplate-depth", so I'm stuck with what the default there allows me. I haven't seen a problem like yours, but I DID run into the template limit.

If you have to increase the setting to over 4000, then clearly you have a LOT of depth, and that's probably creating some exponential situation which kills your compile time. I have not really seen an issue with compile time, but my limit came from VS, and is below 400, I think.

When I ran into the template-depth problem I solved it using a "template firewall" using custom parsers (very few) to solve small tricky common cases, like an identifier parser (plus, I parse tokens, not characters, so I save some depth by the help of the lexer and the custom basic token parser I use for everything). I also have a custom parser for parsing a whole expression (that also cleans up the ast a bit before returning it), so that my grammars that contain an expression doesn't add to the template depth. The real "firewalling" comes, of course, from calling a non-template function to do the parsing (notice the call to parse_expression in the templated parser), like so:

In expression.h:

  struct expression_class;
  using expression_type = x3::rule<expression_class, ast::Expression>;
  BOOST_SPIRIT_DECLARE(expression_type);
  const expression_type& expression_rule();

  bool parse_expression(iterator_type& iIter, const iterator_type& iEnd, error_handler_type& iErrHandler, ast::Expression& oAst);

  struct OptimizedExpressionParser : x3::parser<OptimizedExpressionParser> {  // Parse an expression and return an optimized ast
    using attribute_type = ast::Expression;
    static bool const has_attribute = true;

    template <typename Ctx, typename Attribute>
    bool parse(iterator_type& iFirst, const iterator_type& iLast, const Ctx& iCtx, x3::unused_type, Attribute& oAttr) const {
      ast::Expression a;
      if (parse(iFirst, iLast, iCtx, x3::unused, a)) {
        x3::traits::move_to(a, oAttr);
        return true;
      }
      return false;
    }

    template <typename Ctx>
    bool parse(iterator_type& iFirst, const iterator_type& iLast, const Ctx& iCtx, x3::unused_type, ast::Expression& oAttr) const {
      if (iFirst != iLast) {
        auto& e = x3::get<error_handler_tag>(iCtx).get();
        if (parse_expression(iFirst, iLast, e, oAttr)) {
          oAttr.optimize();
          return true;
        }
      }
      return false;
    }
  };

In expression.cpp:

  // template firewall function
  bool parse_expression(iterator_type& iIter, const iterator_type& iEnd, error_handler_type& iErrHandler, ast::Expression& oAst) {
    const auto expr = expression_rule();
    auto const exprParser = x3::with<error_handler_tag>(std::ref(iErrHandler))[expr];
    return x3::parse(iIter, iEnd, exprParser, oAst);
  }

In expression_def.h:

  // full expression grammar, plus this:
  const expression_type             expression = "expression";
  const expression_type& expression_rule() {
    return expression;
  }

And used in other _def files like this (TL calls my custom token literal parser):

    const auto expression = OptimizedExpressionParser{};
    const auto assignment = expression >> TL(tokenAssign) >> expression;


I hope I have given you some ideas that may help.... :)

  /Mikael


-----Original Message-----
From: Exagon [mailto:[hidden email]]
Sent: Sunday, October 23, 2016 21:40
To: [hidden email]
Subject: [Spirit-general] Horrible compiletimes and memory usage while compiling a parser with X3

I am currently implementing expressions and the operator hierarchy for my DSL, using boost spirit X3.
I think my parser is semanticaly correct, but when I try to compile it, while compiling, gcc and clang having HUGE memory footprint, compiles for an infinite amount of time, and then quits with "g++: internal compiler error:
Killed (program cc1plus)".
I tried to minimize the code arround the expression parser, but its somehow not that trivial.
Here <http://melpon.org/wandbox/permlink/8J3DNGd8xDy2v6Eo>   is an example
of what causes the compiler error. I dont know if this is a bug, or if I have a semantic bug in my code, or if this is a bug.

I think the problem is somwhere there:

    auto const idx = as<ast::Operation>(helper::idxaccess_op > expression > ']');
    auto const func = as<ast::Operation>(helper::func_call_op > expression%',' > ')');
    auto const data = as<ast::Operation>(helper::access_op > expression);

    auto const func_call_expr_def =
        primary_expr >> *(idx|func|data);

if I change (idx|func|data) to (idx|func), it also compiles forever and uses up to 16GB of ram, but gcc is able to compile it, and the parser works how it should work.

Thanks
Exagon



--
View this message in context: http://boost.2283326.n4.nabble.com/Horrible-compiletimes-and-memory-usage-while-compiling-a-parser-with-X3-tp4689104.html
Sent from the spirit-general mailing list archive at Nabble.com.

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, SlashDot.org! http://sdm.link/slashdot
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Exagon
@Mikael Asplund
Thank you very much for your explanation, I think I will try the thing with the custom parser and the non templated parse function.
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

sehe
In reply to this post by Jens Kallup
On 23-10-16 22:48, Jens Kallup wrote:
> No, You definitive #include to many headers in the sources.
> Try/Find out what You need, and delete not used.
Reducing includes helps, and is a recommended approach to reduce Spirit
Qi compile times

> BTW:
>
> #include <boost/config/warning_disable.hpp>
> #include <boost/spirit/include/qi.hpp>
> #include <boost/spirit/include/phoenix.hpp>
Those aren't X3 headers, and they are the "include all" convenience
headers in Qi - the opposite of being selective

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

sehe
In reply to this post by Jens Kallup
On 24-10-16 00:01, Jens Kallup wrote:
> I recommend to use boost version.
> I can't say, who maintain it, but I had no concerns.

You're on the spirit mailing list. The developers and maintainers of all
versions, including X3 are here


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

sehe
In reply to this post by Exagon
On 23-10-16 21:40, Exagon wrote:
I tried to minimize the code arround the expression parser, but its somehow
not that trivial. 

I've noticed similar issues with much simpler code: https://stackoverflow.com/questions/37230653/compile-times-with-boost-spirit-x3/37242969#37242969

Quoting my own comments there

> @RichardHodges This timing though is certainly indicative of a bug/implementation flaw. Usual X3 grammars this would compile in seconds. I'd expect 10s maybe due to the other uses of variants (apply_visitor) which are quite template heavy. But as you can see in my answer, the situation is a lot more severe than this. – sehe May 15 at 19:52


Do any of our X3 buffs know what is going on?

I personally think it's instantiation depth in the face of aggressively inlining compilers. (It's hard to decide what is the best approach. We want inlining, but at what cost? Is there a good, library-defined, cut-off point?)

Seth


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Exagon
I am sorry I wasn't able to minimize the code more than my example.
The approach of creating an custom parser with a parse function which calls a non template parse function works well for me in this example.
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

sehe
On 25-10-16 23:52, Exagon wrote:
> I am sorry I wasn't able to minimize the code more than my example.
> The approach of creating an custom parser with a parse function which calls
> a non template parse function works well for me in this example.

Oh, I didn't think your sample was too large!

I was merely pointing out that there appears to be a real design issue
with X3. The library was intended to resolve compilation time issues of
Qi, not exarcabate them.

And I posted that simpler case because it will be easier to use in
spotting the things that give compilers a hard time.

Whether or not you have a workaround, my feeling is this might need to
be addressed in X3.

Cheers,

Seth


------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Exagon
This post was updated on .
When someone wonders how the workaround suggested by Mikael Asplund looks,here it is for my example.
I am having this problem te second time with spirit x3, but this time I decided to say something about, because it was much worse than the first time.
I hope this can get fixed in the future.
greetings
Exagon
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Mikael Asplund
Did you manage to get your problem down? How much?

  /Mikael

-----Original Message-----
From: Exagon [mailto:[hidden email]]
Sent: Wednesday, October 26, 2016 01:14
To: [hidden email]
Subject: Re: [Spirit-general] Horrible compiletimes and memory usage while compiling a parser with X3

When someone wonders how the workaround suggested by Mikael Asplund looks,
here <http://melpon.org/wandbox/permlink/uz9Ff5EEw5Qa1WxJ>   it is for my
example.
greetings
Exagon




--
View this message in context: http://boost.2283326.n4.nabble.com/Horrible-compiletimes-and-memory-usage-while-compiling-a-parser-with-X3-tp4689104p4689215.html
Sent from the spirit-general mailing list archive at Nabble.com.

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general

------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Exagon
This post was updated on .
Quite a lot, using your idea, the compiletimes go down from 1h to 2-5 min.
Also the memory usage while compiling goes down to maybe 2gb, which is very nice.
I had to play a little bit arround to make my parser and the parse function work with skippers etc. but in the end this works and it performs realy well.
I dont have any results how the parser which is generated using your workarround performs vs the parser which compiles 1h, maybe i will do a comparison when I have enough time to do this.
Also the resulting binary is now only 74mb big (without using the -s flag or compiling with -Os) which is much better than the 1.6gb it had before. :)

greetings
Exagon
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

teajay-2
Le 26/10/2016 à 02:07, Exagon a écrit :

> Quite a lot, using your idea, the compiletimes go down from 1h to 2-5 min.
> Also the memory usage while compiling goes down to maybe 2gb, which is very
> nice.
> I had to play a little bit arround to make my parser and the parse function
> work with skippers etc. but in the end this works and it performs realy
> well.
> I dont have any results how the parser which is generated using your
> workarround performs vs the parser which compiles 1h, maybe i will do a
> comparison when I have enough time to do this.
> Also the resulting binary is now only 74mb big (without using the -s flag or
> compiling with -Os) which is much better I think than the 1.6gb it had
> before :)
>
The memory for compiling and the code size you obtain are pretty big and
are not what I would expect for the small code you posted. We had this
kind of problem before ... this kind of has a bad smell. There is
probably a design issue hiding somewhere. I'll try to find some time to
investigate a bit.

Regards,
Thomas Bernard



------------------------------------------------------------------------------
The Command Line: Reinvented for Modern Developers
Did the resurgence of CLI tooling catch you by surprise?
Reconnect with the command line and become more productive.
Learn the new .NET and ASP.NET CLI. Get your free copy!
http://sdm.link/telerik
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

Exagon
of course these are the numbers of my project, which has some more code under the hood, I I dont think the example of the problem will generate a binary as big as the one from my project was, but the source of the bad compile times and huge binarys can definetly be found in the small example I showed you.
Greetings Exagon
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

teajay-2
Le 26/10/2016 à 19:03, Exagon a écrit :
> of course these are the numbers of my project, which has some more code under
> the hood, I I dont think the example of the problem will generate a binary
> as big as the one from my project was, but the source of the bad compile
> times and huge binarys can definetly be found in the small example I showed
> you.
> Greetings Exagon
>

I managed to get the code compiled successfully with a reasonable
compile time and memory usage with the latest clang version. I didn't
have any success with gcc however.

I tried to identify template instantiations loops using templight but
found nothing really bothering.

I guess you ran into a case where spirit drove the compiler over the
edge, or into a bug. This can happen. Sometimes a new compiler helps get
the problem fixed.

Regards,
Thomas Bernard

------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Horrible compiletimes and memory usage while compiling a parser with X3

cppljevans
On 11/02/2016 08:15 PM, teajay wrote:
 > Le 26/10/2016 à 19:03, Exagon a écrit :
 >> of course these are the numbers of my project, which has some more
code under
 >> the hood, I I dont think the example of the problem will generate a
binary
 >> as big as the one from my project was, but the source of the bad compile
 >> times and huge binarys can definetly be found in the small example I
showed
 >> you.
 >> Greetings Exagon
 >>
 >
 > I managed to get the code compiled successfully with a reasonable
 > compile time and memory usage with the latest clang version. I didn't
 > have any success with gcc however.
 >
 > I tried to identify template instantiations loops using templight but
 > found nothing really bothering.
 >
 > I guess you ran into a case where spirit drove the compiler over the
 > edge, or into a bug. This can happen. Sometimes a new compiler helps get
 > the problem fixed.
 >
 > Regards,
 > Thomas Bernard

I'm spitballing here, but maybe a redesign of spirit to use
virtual functions (or dynamic polymorphism) as well as
templates (or static polymorphism) would help.  Currently,
the static polymorphism design is at least one of the causes
for large compile times (and, I would also guess,
hard-to-read compiler error messages).

IIRC (in one of Joel's ancient posts?), the the predecessor
to spirit did use dynamic polymorphism, and, again IIRC, it
was switched to static polymorphism to speed the run-time.
However, the speed-up in run-time came at the cost of
slow-down at compile time.

What I'm proposing is analogous to current compilers
supplying an optimization flag.  For example, -O0 means no
optimization (analogous to the proposed spirit dynamic
polymorphism) and -O3 means strong optimization (analogous
to the proposed spirit static polymorphism == the current
spirit).

Spirit users would develop their grammars using the dynamic
spirit, and once it was debugged, they could switch to the
static spirit.  I would guess that the dynamic spirit could
would have virtual functions to generate the code for the
static spirit. This is analogous to bison taking a parse.yxx
file and parse.tab.cxx.

Of course there is the small matter of the, I would guess,
somewhat large development effort in such a redesign ;)
Whether that development effort is worth the saving in
compile time's and development times of future spirit users
is an open question, but worth consideration.

-regards,
Larry



------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
123