[Bug Sprint] The November bug sprint starts tomorrow!

classic Classic list List threaded Threaded
26 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[Bug Sprint] The November bug sprint starts tomorrow!

Marshall Clow-2
The Boost bug sprint starts tomorrow Saturday, November 27, 2010 and runs through Sunday December 5th.

The goal of the bug sprint is to make Boost better; the way we measure this is to look at the open tickets in the Trac system.
In the last bug sprint (back in June), we closed over 200 tickets.

However, in the past five months, the number of open tickets has crept back up, so that we once again have more than 900 open tickets.
(the count was down under 800 at the end of the previous sprint).

General information about the bug sprint is here: https://svn.boost.org/trac/boost/wiki/BugSprintNov2010
General information about how the Trac system works is here: https://svn.boost.org/trac/boost/wiki/TicketWorkflow

Useful reports:
https://svn.boost.org/trac/boost/report/1                -- full list of tickets
https://svn.boost.org/trac/boost/report/18                -- ticket counts by owner
https://svn.boost.org/trac/boost/report/19                -- ticket counts by component
https://svn.boost.org/trac/boost/report/20                -- ticket counts by ticket type
https://svn.boost.org/trac/boost/report/21                -- ticket counts by milestone

-- Marshall


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Bug Sprint] The November bug sprint starts tomorrow!

Klaim - Joël Lamotte
Hi,
not sure if it's a revelant information but I tried the firs link and got a
server error with this reason : "OperationalError: database is locked"

I just retried and it work correctly now.


On Fri, Nov 26, 2010 at 21:31, Marshall Clow <[hidden email]> wrote:

> The Boost bug sprint starts tomorrow Saturday, November 27, 2010 and runs
> through Sunday December 5th.
>
> The goal of the bug sprint is to make Boost better; the way we measure this
> is to look at the open tickets in the Trac system.
> In the last bug sprint (back in June), we closed over 200 tickets.
>
> However, in the past five months, the number of open tickets has crept back
> up, so that we once again have more than 900 open tickets.
> (the count was down under 800 at the end of the previous sprint).
>
> General information about the bug sprint is here:
> https://svn.boost.org/trac/boost/wiki/BugSprintNov2010
> General information about how the Trac system works is here:
> https://svn.boost.org/trac/boost/wiki/TicketWorkflow
>
> Useful reports:
> https://svn.boost.org/trac/boost/report/1               -- full list of
> tickets
> https://svn.boost.org/trac/boost/report/18              -- ticket counts
> by owner
> https://svn.boost.org/trac/boost/report/19              -- ticket counts
> by component
> https://svn.boost.org/trac/boost/report/20              -- ticket counts
> by ticket type
> https://svn.boost.org/trac/boost/report/21              -- ticket counts
> by milestone
>
> -- Marshall
>
>
> _______________________________________________
> Unsubscribe & other changes:
> http://lists.boost.org/mailman/listinfo.cgi/boost
>
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Bug Sprint] The November bug sprint starts tomorrow!

Rene Rivera-2
On 11/26/2010 4:09 PM, Klaim wrote:
> not sure if it's a revelant information but I tried the firs link and got a
> server error with this reason : "OperationalError: database is locked"
>
> I just retried and it work correctly now.

Yes, I'm aware of, and trying to fix, that problem. It's intermittent,
but recurring with sufficient propensity to be really annoying :-(


--
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Bug Sprint] The November bug sprint starts tomorrow!

Dean Michael Berris
On Sat, Nov 27, 2010 at 6:25 AM, Rene Rivera <[hidden email]> wrote:

> On 11/26/2010 4:09 PM, Klaim wrote:
>>
>> not sure if it's a revelant information but I tried the firs link and got
>> a
>> server error with this reason : "OperationalError: database is locked"
>>
>> I just retried and it work correctly now.
>
> Yes, I'm aware of, and trying to fix, that problem. It's intermittent, but
> recurring with sufficient propensity to be really annoying :-(
>

This is inherent in SQLite because it's designed to be a
single-threaded embedded database. If there's an option to use BDB,
PostgreSQL, or MySQL as the backing store, we can avoid this database
locking issue.

--
Dean Michael Berris
deanberris.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [Bug Sprint] The November bug sprint starts tomorrow!

Vicente Botet
In reply to this post by Marshall Clow-2

Anthony, has this ticket already been solved?
#3628  condition.notify_one() does not wake up a thread that is in condition.wait() or ondition.timed_wait()  

Best,
Vicente
 
Reply | Threaded
Open this post in threaded view
|

Re: [Bug Sprint] The November bug sprint starts tomorrow!

Jim Bell
In reply to this post by Dean Michael Berris


On 1:59 PM, Dean Michael Berris wrote:

> On Sat, Nov 27, 2010 at 6:25 AM, Rene Rivera <[hidden email]> wrote:
>> On 11/26/2010 4:09 PM, Klaim wrote:
>>> not sure if it's a revelant information but I tried the firs link and got
>>> a
>>> server error with this reason : "OperationalError: database is locked"
>>>
>>> I just retried and it work correctly now.
>> Yes, I'm aware of, and trying to fix, that problem. It's intermittent, but
>> recurring with sufficient propensity to be really annoying :-(
>>
> This is inherent in SQLite because it's designed to be a
> single-threaded embedded database. If there's an option to use BDB,
> PostgreSQL, or MySQL as the backing store, we can avoid this database
> locking issue.

Can we tune SQLite?

I've found big performance improvements with some of these pragmas...
PRAGMA journal_mode = OFF;
PRAGMA synchronous = OFF;
PRAGMA cache_size = 20000;
PRAGMA locking_mode = EXCLUSIVE;
PRAGMA temp_store = MEMORY;

You have to have enough control to run them every time the library
initializes. And, of course, understand their consequences.
<http://sqlite.org/pragma.html>


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

[trac] Website performance, and disabling the code browser?

Rene Rivera-2
On 11/28/2010 7:45 AM, Jim Bell wrote:

> On 1:59 PM, Dean Michael Berris wrote:
>> On Sat, Nov 27, 2010 at 6:25 AM, Rene Rivera<[hidden email]>  wrote:
>>> On 11/26/2010 4:09 PM, Klaim wrote:
>>>> not sure if it's a revelant information but I tried the firs link and got
>>>> a
>>>> server error with this reason : "OperationalError: database is locked"
>>>>
>>>> I just retried and it work correctly now.
>>> Yes, I'm aware of, and trying to fix, that problem. It's intermittent, but
>>> recurring with sufficient propensity to be really annoying :-(
>>>
>> This is inherent in SQLite because it's designed to be a
>> single-threaded embedded database. If there's an option to use BDB,
>> PostgreSQL, or MySQL as the backing store, we can avoid this database
>> locking issue.

The idea of switching to MySQL is something I've been investigating this
past week (when I've gotten a chance from work) and it's not something
that can be done easily, AFAICT. It essentially boils down to exporting,
editing SQL, importing, and crossing fingers that Trac understands the
new data.

> Can we tune SQLite?
>
> I've found big performance improvements with some of these pragmas...
> PRAGMA journal_mode = OFF;
> PRAGMA synchronous = OFF;
> PRAGMA cache_size = 20000;
> PRAGMA locking_mode = EXCLUSIVE;
> PRAGMA temp_store = MEMORY;
>
> You have to have enough control to run them every time the library
> initializes. And, of course, understand their consequences.
> <http://sqlite.org/pragma.html>

Not sure we have that control since it's inside Trac code. One item that
the performance measurements have brought is that slowest part is the
Trac code browser. So one possible option is to disable just the browser
component to see if things improve. Of course the drawback of that is
loosing that functionality. Which is rather convenient.. But isn't
something we can't get with other tools, like the direct web access SVN
provides and things like TSVN.

--
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Artyom Beilis
>
> > Can we tune SQLite?
> >
> > I've  found big performance improvements with some of these pragmas...
> > PRAGMA  journal_mode = OFF;

Very BAD - you would not be able to do rollback.

> > PRAGMA synchronous = OFF;

It actually has effect only on writes. So it does
not do complete fsync() after each transaction.

It really makes write transactions much faster but
you may find yourself with corrupted database if something
goes wrong.

Also I don't think that write translations are the bottle
neck of the service.  

I would not recommend you doing such things.

I don't think trac is slow because of database, it is just
slow it is written in Python and it is very slow.

It is quite big misconception that DB is bottle neck.

If some searches are slow you may need indexes switching DB would not help.
but other then that, Sqlite3 is VERY fast and I don't think.

Once I did tests with Wordpress and it could generate 4-5 pages per second
when if I separated all SQL queries and executed them (about 20 per page)
I could run about 20x1000 queries per second. So I would not start
from blaming Sqlite.

Artyom



     
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Dean Michael Berris
On Mon, Nov 29, 2010 at 4:31 AM, Artyom <[hidden email]> wrote:
>
> I don't think trac is slow because of database, it is just
> slow it is written in Python and it is very slow.
>

I don't think the discussion was about Trac being slow. The point was
that "Database Lock" errors happen pretty often now.

> It is quite big misconception that DB is bottle neck.
>
> If some searches are slow you may need indexes switching DB would not help.
> but other then that, Sqlite3 is VERY fast and I don't think.
>

Sorry, but SQLite is not fast enough if you have multiple processes
trying to perform the same query on the same database. The reason is
because of the database locking mechanism that the SQLite library
enforces on the database file.

If you had a single process and multiple threads and serialized all
access to the SQLite DB on a single thread then that would perform
better. Unfortunately Trac wasn't implemented this way and everything
that happens within the WSGI handler is blocking -- also consider that
Boost has thousands of tickets and multiple query types, most of which
have JOIN's IIRC.

SQLite is fast if you had enough memory to put the whole DB in memory,
and only ever performed primary key query lookups. Introduce JOINs and
then you run into trouble.

> Once I did tests with Wordpress and it could generate 4-5 pages per second
> when if I separated all SQL queries and executed them (about 20 per page)
> I could run about 20x1000 queries per second. So I would not start
> from blaming Sqlite.
>

But wordpress doesn't do JOINs and only ever does simple selects and
sorts on indexes. See how Trac does it and you'll see why SQLite is
the wrong choice for any sufficiently large installation.

HTH

--
Dean Michael Berris
deanberris.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Dean Michael Berris
In reply to this post by Rene Rivera-2
On Sun, Nov 28, 2010 at 11:09 PM, Rene Rivera <[hidden email]> wrote:

> On 11/28/2010 7:45 AM, Jim Bell wrote:
>>
>> On 1:59 PM, Dean Michael Berris wrote:
>>>
>>> This is inherent in SQLite because it's designed to be a
>>> single-threaded embedded database. If there's an option to use BDB,
>>> PostgreSQL, or MySQL as the backing store, we can avoid this database
>>> locking issue.
>
> The idea of switching to MySQL is something I've been investigating this
> past week (when I've gotten a chance from work) and it's not something that
> can be done easily, AFAICT. It essentially boils down to exporting, editing
> SQL, importing, and crossing fingers that Trac understands the new data.
>

Interesting. There's no tool or tutorial that comes with the MySQL
storage module for Trac? I'd imagine someone's already done a
migration tool for that. I'll Google around a bit to see if I can find
anything in that line.

>> Can we tune SQLite?
>>
>> I've found big performance improvements with some of these pragmas...
>> PRAGMA journal_mode = OFF;
>> PRAGMA synchronous = OFF;
>> PRAGMA cache_size = 20000;
>> PRAGMA locking_mode = EXCLUSIVE;
>> PRAGMA temp_store = MEMORY;
>>
>> You have to have enough control to run them every time the library
>> initializes. And, of course, understand their consequences.
>> <http://sqlite.org/pragma.html>
>
> Not sure we have that control since it's inside Trac code. One item that the
> performance measurements have brought is that slowest part is the Trac code
> browser. So one possible option is to disable just the browser component to
> see if things improve. Of course the drawback of that is loosing that
> functionality. Which is rather convenient.. But isn't something we can't get
> with other tools, like the direct web access SVN provides and things like
> TSVN.
>

I like the idea of disabling the code browser. I can just imagine
Google or a non-compliant spider crawling that and unintentionally
causing a DoS of the server. ;)

Is there a way for us to put Google Analytics on each page of Trac?
That way we can see which pages are actually visited by people. I'm
not very familiar with the template system but maybe someone else has
already done a Trac module for putting Google Analytics in. I'll
Google around some to see if I can find anything to this effect.

--
Dean Michael Berris
deanberris.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Dean Michael Berris
On Mon, Nov 29, 2010 at 6:35 AM, Dean Michael Berris
<[hidden email]> wrote:

> On Sun, Nov 28, 2010 at 11:09 PM, Rene Rivera <[hidden email]> wrote:
>>
>> The idea of switching to MySQL is something I've been investigating this
>> past week (when I've gotten a chance from work) and it's not something that
>> can be done easily, AFAICT. It essentially boils down to exporting, editing
>> SQL, importing, and crossing fingers that Trac understands the new data.
>>
>
> Interesting. There's no tool or tutorial that comes with the MySQL
> storage module for Trac? I'd imagine someone's already done a
> migration tool for that. I'll Google around a bit to see if I can find
> anything in that line.
>

What I found with a quick Googling is this:
http://trac.edgewall.org/wiki/SqLiteToMySql -- I'm positive you saw
this also while looking around I imagine. :D

>
> Is there a way for us to put Google Analytics on each page of Trac?
> That way we can see which pages are actually visited by people. I'm
> not very familiar with the template system but maybe someone else has
> already done a Trac module for putting Google Analytics in. I'll
> Google around some to see if I can find anything to this effect.
>

And a quick Googling around yielded this:
http://google.ufsoft.org/wiki/TracGoogleAnalytics -- maybe something
we can play around with to get a better idea what people are actually
doing on the Trac site. :)

HTH

--
Dean Michael Berris
deanberris.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Michael Caisse-2
In reply to this post by Dean Michael Berris
On 11/28/2010 2:30 PM, Dean Michael Berris wrote:

> On Mon, Nov 29, 2010 at 4:31 AM, Artyom<[hidden email]>  wrote:
>> I don't think trac is slow because of database, it is just
>> slow it is written in Python and it is very slow.
>>
> I don't think the discussion was about Trac being slow. The point was
> that "Database Lock" errors happen pretty often now.
>
>> It is quite big misconception that DB is bottle neck.
>>
>> If some searches are slow you may need indexes switching DB would not help.
>> but other then that, Sqlite3 is VERY fast and I don't think.
>>
> Sorry, but SQLite is not fast enough if you have multiple processes
> trying to perform the same query on the same database. The reason is
> because of the database locking mechanism that the SQLite library
> enforces on the database file.
>

Dean -

Thanks for debunking Artyom's notions. I can attest from a python
project I'm in the middle of right now that the sqlite performance is
abysmal except in the simple cases. With the postgresql back end it runs
"speedy fast".

Rene / Dean -

If there is something I can do to help migrate to a different backend
just let me know.


michael

--

----------------------------------
Michael Caisse
Object Modeling Designs
www.objectmodelingdesigns.com


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Belcourt, Kenneth

On Nov 28, 2010, at 4:07 PM, Michael Caisse wrote:

> On 11/28/2010 2:30 PM, Dean Michael Berris wrote:
>> On Mon, Nov 29, 2010 at 4:31 AM, Artyom<[hidden email]>  wrote:
>>> I don't think trac is slow because of database, it is just
>>> slow it is written in Python and it is very slow.
>>>
>> I don't think the discussion was about Trac being slow. The point was
>> that "Database Lock" errors happen pretty often now.

And it's covariant form:

Safari can’t open the page “http://www.boost.org/development/tests/trunk/developer/summary.html 
” because the server where this page is located isn’t responding.

I haven't been following the server performance discussion too much  
but could we in some fashion separate the web page server from the  
trac server?  I'm afraid that the large increase in the number of  
trunk and release testers combined with a steady increase in the  
number of Boost libraries and tests may be partly to blame.  One way  
to test this is to reduce the number of testers (and delete their  
results) and see if there's any noticeable improvement in server  
performance.  Is this worth trying?

-- Noel


_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Phil Endecott-48
In reply to this post by Dean Michael Berris
Dean Michael Berris wrote:
> On Mon, Nov 29, 2010 at 4:31 AM, Artyom <[hidden email]> wrote:
>> It is quite big misconception that DB is bottle neck.
>
> Sorry, but SQLite is not fast enough if you have multiple processes
> trying to perform the same query on the same database. The reason is
> because of the database locking mechanism that the SQLite library
> enforces on the database file.

My understanding was that SQLite only did locking for writes, i.e. it's
possible for multiple threads to be reading at the same time.  In fact this
is FAQ #5 at http://www.sqlite.org/faq.html#q5 :

     "SQLite allows multiple processes to have the database file open at
     once, and for multiple processes to read the database at once. When
     any process wants to write, it must lock the entire database file
     for the duration of its update. But that normally only takes a few
     milliseconds. Other processes just wait on the writer to finish then
     continue about their business."

Is trac doing writes for things that should only be reads, perhaps?


(in another post:)
> I like the idea of disabling the code browser.

I don't; to me an occasional error message is better than not working
at all.


Phil.




_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Dean Michael Berris
On Mon, Nov 29, 2010 at 8:09 AM, Phil Endecott
<[hidden email]> wrote:

> Dean Michael Berris wrote:
>>
>> On Mon, Nov 29, 2010 at 4:31 AM, Artyom <[hidden email]> wrote:
>>>
>>> It is quite big misconception that DB is bottle neck.
>>
>> Sorry, but SQLite is not fast enough if you have multiple processes
>> trying to perform the same query on the same database. The reason is
>> because of the database locking mechanism that the SQLite library
>> enforces on the database file.
>
> My understanding was that SQLite only did locking for writes, i.e. it's
> possible for multiple threads to be reading at the same time.  In fact this
> is FAQ #5 at http://www.sqlite.org/faq.html#q5 :
>
>    "SQLite allows multiple processes to have the database file open at
>    once, and for multiple processes to read the database at once. When
>    any process wants to write, it must lock the entire database file
>    for the duration of its update. But that normally only takes a few
>    milliseconds. Other processes just wait on the writer to finish then
>    continue about their business."
>
> Is trac doing writes for things that should only be reads, perhaps?
>

What I understand is that the Python implementation is not only
inefficient this way, because it just doesn't leave the DB locking to
the C library. If you look at the relevant part in the Python SQLite
DB API adapter implementation:

"When a database is accessed by multiple connections, and one of the
processes modifies the database, the SQLite database is locked until
that transaction is committed. The timeout parameter specifies how
long the connection should wait for the lock to go away until raising
an exception. The default for the timeout parameter is 5.0 (five
seconds)."

Note that a COMMIT has to succeed in order for the other reader
processes to get a chance to perform the read -- also writes on large
enough SQLite DB's that are normalized (i.e. having a lot of tables
and performing joins with lots of integrity constraints) means that
your other processes can wait in the order of seconds -- especially
when you're also serving static web pages and generating dynamic
pages, with a disk spinning at 5400 or even 7200 rpm. The reason for
this is SQLite's design which uses a single file for the whole
database. Deletions are particularly hairy, and when doing joins to
yield thousands of rows (as in the case of Boost), you run into a
perfect storm.

Especially now that we're doing a bug sprint, I don't see SQLite being
scalable compared to say MySQL/PgSQL which are optimized to do
compute-intensive and highly concurrent queries.

>
> (in another post:)
>>
>> I like the idea of disabling the code browser.
>
> I don't; to me an occasional error message is better than not working at
> all.
>

Not working at all -- you mean the code browser right? I think Trac's
value right now is really the issue tracking and the Wiki, not
necessarily the code browser.

It should be trivial so serve the Subversion repository -- or a mirror
of it -- for online viewing without having to use Trac's SVN browser.

--
Dean Michael Berris
deanberris.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Rene Rivera-2
On 11/28/2010 7:07 PM, Dean Michael Berris wrote:
> On Mon, Nov 29, 2010 at 8:09 AM, Phil Endecott
> <[hidden email]>  wrote:
>> Dean Michael Berris wrote:

[cut db performance stuff that I can't really do anything about right now]

>> (in another post:)
>>>
>>> I like the idea of disabling the code browser.
>>
>> I don't; to me an occasional error message is better than not working at
>> all.
>>
>
> Not working at all -- you mean the code browser right? I think Trac's
> value right now is really the issue tracking and the Wiki, not
> necessarily the code browser.
>
> It should be trivial so serve the Subversion repository -- or a mirror
> of it -- for online viewing without having to use Trac's SVN browser.

The SVN tree is already available for browsing from SVN directly. Just
go to <http://svn.boost.org/svn/boost/>.


--
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Rene Rivera-2
In reply to this post by Dean Michael Berris
On 11/28/2010 4:47 PM, Dean Michael Berris wrote:

> On Mon, Nov 29, 2010 at 6:35 AM, Dean Michael Berris
> <[hidden email]>  wrote:
>> On Sun, Nov 28, 2010 at 11:09 PM, Rene Rivera<[hidden email]>  wrote:
>>>
>>> The idea of switching to MySQL is something I've been investigating this
>>> past week (when I've gotten a chance from work) and it's not something that
>>> can be done easily, AFAICT. It essentially boils down to exporting, editing
>>> SQL, importing, and crossing fingers that Trac understands the new data.
>>>
>>
>> Interesting. There's no tool or tutorial that comes with the MySQL
>> storage module for Trac? I'd imagine someone's already done a
>> migration tool for that. I'll Google around a bit to see if I can find
>> anything in that line.
>>
>
> What I found with a quick Googling is this:
> http://trac.edgewall.org/wiki/SqLiteToMySql -- I'm positive you saw
> this also while looking around I imagine. :D

Yep, I've seen that. And the fact that it starts out by requiring an
existing Trac install already running MySQL to get the schema seems
weird to me. Even if it is understandable why. But the real problems
with the Trac migrations I found is that none of them have been tested
against the latest version (what we have).

>> Is there a way for us to put Google Analytics on each page of Trac?
>> That way we can see which pages are actually visited by people. I'm
>> not very familiar with the template system but maybe someone else has
>> already done a Trac module for putting Google Analytics in. I'll
>> Google around some to see if I can find anything to this effect.
>>
>
> And a quick Googling around yielded this:
> http://google.ufsoft.org/wiki/TracGoogleAnalytics -- maybe something
> we can play around with to get a better idea what people are actually
> doing on the Trac site. :)

Not sure what that gains us. And in case I haven't posted this:

(%,seconds,path)
66.79  2450119482.93 /trac
66.27  2431033394.93 /trac/boost
19.24   705711654.42 /trac/boost/browser
18.91   693655902.26 /svn
18.91   693655839.11 /svn/boost
17.72   650181710.98 /svn/boost/!svn
17.07   626102510.20 /trac/boost/changeset
16.88   619327958.22 /svn/boost/!svn/vcc/default
16.88   619327958.22 /svn/boost/!svn/vcc
  6.53   239583046.05 /doc
  6.40   234697665.67 /trac/boost/browser/sandbox


--
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Dean Michael Berris
On Mon, Nov 29, 2010 at 11:01 AM, Rene Rivera <[hidden email]> wrote:

> On 11/28/2010 4:47 PM, Dean Michael Berris wrote:
>>
>> What I found with a quick Googling is this:
>> http://trac.edgewall.org/wiki/SqLiteToMySql -- I'm positive you saw
>> this also while looking around I imagine. :D
>
> Yep, I've seen that. And the fact that it starts out by requiring an
> existing Trac install already running MySQL to get the schema seems weird to
> me. Even if it is understandable why. But the real problems with the Trac
> migrations I found is that none of them have been tested against the latest
> version (what we have).
>

Yeah, I was trying to look for those that worked with 0.12 and I
wasn't finding much luck with that.

>>> Is there a way for us to put Google Analytics on each page of Trac?
>>> That way we can see which pages are actually visited by people. I'm
>>> not very familiar with the template system but maybe someone else has
>>> already done a Trac module for putting Google Analytics in. I'll
>>> Google around some to see if I can find anything to this effect.
>>>
>>
>> And a quick Googling around yielded this:
>> http://google.ufsoft.org/wiki/TracGoogleAnalytics -- maybe something
>> we can play around with to get a better idea what people are actually
>> doing on the Trac site. :)
>
> Not sure what that gains us. And in case I haven't posted this:
>

Well, it gives us a better idea as to things like time of day, where
the traffic is coming from, what pages they're looking at, etc. -- all
the good things that Google Analytics gives us for free.

Nonetheless the report breakdown below looks interesting.

> (%,seconds,path)
> 66.79  2450119482.93 /trac
> 66.27  2431033394.93 /trac/boost
> 19.24   705711654.42 /trac/boost/browser
> 18.91   693655902.26 /svn
> 18.91   693655839.11 /svn/boost
> 17.72   650181710.98 /svn/boost/!svn
> 17.07   626102510.20 /trac/boost/changeset
> 16.88   619327958.22 /svn/boost/!svn/vcc/default
> 16.88   619327958.22 /svn/boost/!svn/vcc
>  6.53   239583046.05 /doc
>  6.40   234697665.67 /trac/boost/browser/sandbox
>

This is % of what, time since, ever? Or time since a given start date?
Do we have ideas as to the daily traffic the trac site gets?

Also it looks like the browser is up at the top 3 most visited. Either
we disable that or enable caching the pages if that's even possible,
maybe look at putting varnish [0] in front of the Apache server. This
will require though that we lose the SSL for Trac, I'm not sure if
that's something people would be amenable to.

[0] -- http://www.varnish-cache.org/

Is the Trac Wiki hosted on SQLite too? Or is the Wiki storage system
external and doesn't touch the SQLite database?

Thanks Rene for taking time to engage in discourse here.

--
Dean Michael Berris
deanberris.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Rene Rivera-2
On 11/28/2010 9:29 PM, Dean Michael Berris wrote:
> On Mon, Nov 29, 2010 at 11:01 AM, Rene Rivera<[hidden email]>  wrote:

> Nonetheless the report breakdown below looks interesting.
>
>> (%,seconds,path)
>> 66.79  2450119482.93 /trac
>> 66.27  2431033394.93 /trac/boost
>> 19.24   705711654.42 /trac/boost/browser
>> 18.91   693655902.26 /svn
>> 18.91   693655839.11 /svn/boost
>> 17.72   650181710.98 /svn/boost/!svn
>> 17.07   626102510.20 /trac/boost/changeset
>> 16.88   619327958.22 /svn/boost/!svn/vcc/default
>> 16.88   619327958.22 /svn/boost/!svn/vcc
>>   6.53   239583046.05 /doc
>>   6.40   234697665.67 /trac/boost/browser/sandbox
>>
>
> This is % of what, time since, ever? Or time since a given start date?
> Do we have ideas as to the daily traffic the trac site gets?

It's % of total time (cumulative). So "/trac" is taking up %67 of the
total time people waited for the page to get served overall and so on
down the tree.

The time is for the week ending on the 21st. I haven't done the report
for this past week as it's likely not going to be different and it takes
some time to process the 300M of data for it.

The perf stats we keep don't include time stamps, just time spent, so I
can't do daily analysis.

> Also it looks like the browser is up at the top 3 most visited. Either
> we disable that or enable caching the pages if that's even possible,
> maybe look at putting varnish [0] in front of the Apache server. This
> will require though that we lose the SSL for Trac, I'm not sure if
> that's something people would be amenable to.
>
> [0] -- http://www.varnish-cache.org/

It would also require that *all* of the Boost web services be switch to
using a cache proxy. As they all currently live on the same server, and
you can't share ports.

> Is the Trac Wiki hosted on SQLite too? Or is the Wiki storage system
> external and doesn't touch the SQLite database?

The entire Trac DB is SQLite.. which is annoying since one item I
noticed is that it also means cookie storage is in SQLite and many of
the stack traces I've seen when the database lock error happens is in
the cookie reading/writing.

> Thanks Rene for taking time to engage in discourse here.

Welcome.. Just wish there was a good solution to this problem. But I'm
feeling more, and more, that it's a problem of bad database design on
the part of Trac that's the real problem.

--
-- Grafik - Don't Assume Anything
-- Redshift Software, Inc. - http://redshift-software.com
-- rrivera/acm.org (msn) - grafik/redshift-software.com
-- 102708583/icq - grafikrobot/aim,yahoo,skype,efnet,gmail
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
Reply | Threaded
Open this post in threaded view
|

Re: [trac] Website performance, and disabling the code browser?

Dean Michael Berris
On Wed, Dec 1, 2010 at 1:26 AM, Rene Rivera <[hidden email]> wrote:
> On 11/28/2010 9:29 PM, Dean Michael Berris wrote:
>>
>> This is % of what, time since, ever? Or time since a given start date?
>> Do we have ideas as to the daily traffic the trac site gets?
>
> It's % of total time (cumulative). So "/trac" is taking up %67 of the total
> time people waited for the page to get served overall and so on down the
> tree.
>

Alright, that makes sense since /trac is the default page -- does this
mean ticket pages get counted under /trac as well? Or individual
tickets have different entries (I'm assuming this is broken down by
full URI)?

> The time is for the week ending on the 21st. I haven't done the report for
> this past week as it's likely not going to be different and it takes some
> time to process the 300M of data for it.
>
> The perf stats we keep don't include time stamps, just time spent, so I
> can't do daily analysis.
>

Alright. The Google Analytics makes sense for getting this kind of
data though, is as close to real time as possible and is free.

>> Also it looks like the browser is up at the top 3 most visited. Either
>> we disable that or enable caching the pages if that's even possible,
>> maybe look at putting varnish [0] in front of the Apache server. This
>> will require though that we lose the SSL for Trac, I'm not sure if
>> that's something people would be amenable to.
>>
>> [0] -- http://www.varnish-cache.org/
>
> It would also require that *all* of the Boost web services be switch to
> using a cache proxy. As they all currently live on the same server, and you
> can't share ports.
>

That's possible, Varnish is configurable that way -- everything that
goes through the non-SSL site can be cached with Varnish. Although the
biggest down side is that it doesn't work with SSL so something like
Squid might have to be used for reverse-proxying SSL traffic.

>> Is the Trac Wiki hosted on SQLite too? Or is the Wiki storage system
>> external and doesn't touch the SQLite database?
>
> The entire Trac DB is SQLite.. which is annoying since one item I noticed is
> that it also means cookie storage is in SQLite and many of the stack traces
> I've seen when the database lock error happens is in the cookie
> reading/writing.
>

Oh boy. Then because of that reason alone I'd say it'd be better if we
can do scheduled maintenance to try and move to MySQL.

>> Thanks Rene for taking time to engage in discourse here.
>
> Welcome.. Just wish there was a good solution to this problem. But I'm
> feeling more, and more, that it's a problem of bad database design on the
> part of Trac that's the real problem.
>

Yeah, using SQLite might make sense for internal teams, but
public-facing sites that get spammers like Boost does, it doesn't
really work well. :)

--
Dean Michael Berris
deanberris.com
_______________________________________________
Unsubscribe & other changes: http://lists.boost.org/mailman/listinfo.cgi/boost
12