Newbie question - how to improve parsing time with spirit

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Newbie question - how to improve parsing time with spirit

Rajan, Vivek K

Hello-

 

I am parsing large ASCII files (>300-500 meg). I need to extract specific tokens and then values associated with the tokens. I am using the file_parser example from the spirit example distribution. Am noticing slow (>3 minutes) run-times when parsing large files. Any suggestions/feedback for how to improve the speed of parsing? Code snipped is included below.

 

Vivek

 

 

////////////////////////////////////////////////////////////////////////////

//

//  Actions

//

////////////////////////////////////////////////////////////////////////////

void echo(iterator_t first, iterator_t const& last)

{

  while (first != last)

    std::cout << *first++;

}

 

////////////////////////////////////////////////////////////////////////////

//

//  Main program

//

////////////////////////////////////////////////////////////////////////////

int main(int argc, char* argv[])

{

  if (2 > argc) {

    std::cout << "Must specify a filename!\n";

    return -1;

  }

 

  // Create a file iterator for this file

  iterator_t first(argv[1]);

 

  if (!first) {

    std::cout << "Unable to open file!\n";

    return -1;

  }

 

  // Create an EOF iterator

  iterator_t last = first.make_end();

 

  // A simple rule

  rule_t r = *(anychar_p);

 

  // Parse

  parse_info <iterator_t> info = parse(

    first,

    last,

    r[&echo]

  );

 

  // This really shouldn't fail...

  if (info.full)

    std::cout << "Parse succeeded!\n";

  else

    std::cout << "Parse failed!\n";

 

  return 0;

}

Reply | Threaded
Open this post in threaded view
|

Re: Newbie question - how to improve parsing time with spirit

Joel de Guzman-2
Rajan, Vivek K wrote:

> Hello-
>
>  
>
> I am parsing large ASCII files (>300-500 meg). I need to extract
> specific tokens and then values associated with the tokens. I am using
> the file_parser example from the spirit example distribution. Am
> noticing slow (>3 minutes) run-times when parsing large files. Any
> suggestions/feedback for how to improve the speed of parsing? Code
> snipped is included below.

First step would be to isolate the effects of the file_iterator.
You can do this by storing the file in a temp memory buffer
and parsing from there. Have you tried that? I'd be interested
to see how fast that is in compaison. I remember using file_iterator
in quickdoc/quickbook initially and later switched to an all in-
memory storage because of poor performance.

Cheers,
--
Joel de Guzman
http://www.boost-consulting.com
http://spirit.sf.net



-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general
Reply | Threaded
Open this post in threaded view
|

Re: Newbie question - how to improve parsing time with spirit

Stephen Torri
In reply to this post by Rajan, Vivek K
The action 'echo' will add a lot of time on its own for writing every
line. I did a sample program for class which reported each file it found
in a depth-first search of a directory. With IO it took >5 minutes when
starting from /. Without IO it took 30 seconds.

Stephen

On Mon, 2005-09-19 at 16:32 -0700, Rajan, Vivek K wrote:

> ////////////////////////////////////////////////////////////////////////////
>
> //
>
> //  Actions
>
> //
>
> ////////////////////////////////////////////////////////////////////////////
>
> void echo(iterator_t first, iterator_t const& last)
>
> {
>
>   while (first != last)
>
>     std::cout << *first++;
>
> }


signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Newbie question - how to improve parsing time with spirit

Joel de Guzman-2
Stephen Torri wrote:
> The action 'echo' will add a lot of time on its own for writing every
> line. I did a sample program for class which reported each file it found
> in a depth-first search of a directory. With IO it took >5 minutes when
> starting from /. Without IO it took 30 seconds.

Oh yeah! I didn't even notice that std::cout. Of course it will.

Regards,
--
Joel de Guzman
http://www.boost-consulting.com
http://spirit.sf.net



-------------------------------------------------------
SF.Net email is sponsored by:
Tame your development challenges with Apache's Geronimo App Server. Download
it for free - -and be entered to win a 42" plasma tv or your very own
Sony(tm)PSP.  Click here to play: http://sourceforge.net/geronimo.php
_______________________________________________
Spirit-general mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/spirit-general