PHP Crashes and Browser freezes while searching big folder

[rcubetrac]

Reported by altex on 17 Apr 2008 13:11 UTC as Trac ticket #1484994

Hi dear developers. First of all i want to thank you, RC is the best webmail client.

I installed RoundCube for use with gmail account through imap. My Gmail/All Mail folder consists about 31854 messages (>1Gb used space). When i use Search by this big folder my browser totally freezes.

I use

• Fedora 8 linux on OpenVZ container
• apache >= 2.2 (latest available in updates repository)
• Mysql >= 5 (latest available in updates repository)
• php >= 5.2 (latest available in updates repository)
• php memory limit - 64Mb (setted in htaccess)
• Firefox 2.0.0.14

In php logs there is an error, that php is failed to allocate 2048 bites (64Mb used).
So please explain me why 64 Mb is too small to successfully complete search? Is RC trying to read all message's headers in memory while searching? 0_o

Thank You!

Keywords: search crash
Migrated-From: http://trac.roundcube.net/ticket/1484994

[rcubetrac]

Comment by altex on 17 Apr 2008 16:14 UTC

Error was

PHP Fatal error: Allowed memory size of 67108864 bytes exhausted (tried to allocate 2049 bytes) in /var/vhosts/mail/www.secure/program/lib/imap.inc on line 170

[rcubetrac]

Comment by altex on 17 Apr 2008 16:19 UTC

Linux 2.6.18-53.1.6.el5.028stab053.6 #1 SMP
PHP - 5.2.4
Server API - Apache 2.0 Handler
Apache/2.2.8 (Fedora)
MySQL - 5.0.45

[rcubetrac]

Comment by @thomascube on 17 Apr 2008 17:52 UTC

No !RoundCube does not load all headers in memory, just the message index. Headers are then fetched for the number of messages you configured in "rows per page". Looks like our IMAP library missed the end of the response or the size of the result set washuge. What was the expected number of results of your search?

I can search a folder with 15'000 messages in about a second...

[rcubetrac]

Owner changed by @thomascube on 17 Apr 2008 17:52 UTC

=> none

[rcubetrac]

Severity changed by @thomascube on 17 Apr 2008 17:52 UTC

major => normal

[rcubetrac]

Comment by altex on 17 Apr 2008 19:35 UTC

Replying to thomasb:

No !RoundCube does not load all headers in memory, just the message index. Headers are then fetched for the number of messages you configured in "rows per page". Looks like our IMAP library missed the end of the response or the size of the result set washuge. What was the expected number of results of your search?

I can search a folder with 15'000 messages in about a second...

I trying to search small number of documents in this 30000+ folder, so a search the word "" and i found 15 matches within 4 seconds. But when the result matches more than hundreds i get low memory error.

[rcubetrac]

Milestone changed by @alecpl on 11 May 2008 17:13 UTC

later => 0.1.2

[rcubetrac]

Comment by chugadie on 10 Sep 2008 13:37 UTC

It's easy to reproduce this behavior by using htaccess files to limit the memory to 1M and then searching.

To make the system not feel like it's "freezing", the ajax command should be ASYNC

in app.js add "true" as a 3rd parameter to open()

3716 this.xmlhttp.onreadystatechange = function(){ _ref.xmlhttp_onreadystatechange(); };
3717 this.xmlhttp.open('GET', url, true); <-- async flag
3718 this.xmlhttp.setRequestHeader('X-RoundCube-Referer', bw.get_cookie('roundcube_sessid'));
3719 this.xmlhttp.send(null);
3720 };

Also, to gracefully recover from this error, we could add a nice error box to "http_error" in app.js

3550 request_obj.reset();
3551 request_obj.__lock = false;
3552 this.display_message('Unknown Serer Error!', 'error'); <-- nice error box
3553 };

This should at least make RC look like it's not completely non-functional when the server encounters some unexpected error. I can happily change my memory back up and continue using the system.

[rcubetrac]

Comment by chugadie on 10 Sep 2008 13:38 UTC

fix code formatting for the previous comment

in app.js add "true" as a 3rd parameter to open()

  3716     this.xmlhttp.onreadystatechange = function(){ _ref.xmlhttp_onreadystatechange(); };
  3717     this.xmlhttp.open('GET', url, true);
  3718     this.xmlhttp.setRequestHeader('X-RoundCube-Referer', bw.get_cookie('roundcube_sessid'));
  3719     this.xmlhttp.send(null);
  3720     };

Also, to gracefully recover from this error, we could add a nice error box to "http_error" in app.js

  3550     request_obj.reset();
  3551     request_obj.__lock = false;
  3552     this.display_message('Unknown Serer Error!', 'error');
  3553     };
  3554

[rcubetrac]

Comment by @thomascube on 10 Sep 2008 14:43 UTC

Applied the suggested patch in f936501

[rcubetrac]

Comment by @alecpl on 12 Oct 2008 17:50 UTC

To altex, there are some fixes in imap library in svn-trunk version which may fix your issue. Please, try it.

[rcubetrac]

Comment by @alecpl on 3 Nov 2008 12:34 UTC

Old, no feedback.

[rcubetrac]

Status changed by @alecpl on 3 Nov 2008 12:34 UTC

new => closed

[rcubetrac]

Comment by altex on 8 Dec 2010 06:26 UTC

Just updated to 0.4.2-stable.

Unfortunetly i can not check if search work correctly, because "All mail" folder just can not be shown (it's always "loading ..."). It just waits for a couple of minutes and then "loading ...." disapeares, but no mail is shown instead.

I have 26449 letters in "All mail" folder (1.8 Gb).

[rcubetrac]

Status changed by altex on 8 Dec 2010 06:26 UTC

closed => reopened

[rcubetrac]

Comment by altex on 8 Dec 2010 06:27 UTC

there is no errors in error log, and even nothing in access log, so i think php is just been killed by timeout

[rcubetrac]

Comment by altex on 8 Dec 2010 06:33 UTC

and i have 13703 unread letters in "All mail" folder

[rcubetrac]

Comment by @alecpl on 8 Dec 2010 07:09 UTC

Check Roundcube log. Enable imap_debug and attach the log.

[rcubetrac]

Milestone changed by @alecpl on 8 Dec 2010 07:09 UTC

0.2-stable => 0.5-stable

[rcubetrac]

Comment by altex on 8 Dec 2010 11:19 UTC

It was "Request timed out" in top popup box after "Loading...".

Here is the log file

[rcubetrac]

Comment by altex on 8 Dec 2010 11:20 UTC

cant upload file
Error: Upload failed
Maximum attachment size: 262144 bytes

Its size is 1 959 719 bytes.

[rcubetrac]

Comment by @alecpl on 8 Dec 2010 11:30 UTC

Use (g)zip.

[rcubetrac]

Comment by @alecpl on 8 Dec 2010 11:33 UTC

Request timeout is 180 sec. (app.js) So, maybe your IMAP server just doesn't finish a response in this time.

[rcubetrac]

Comment by altex on 8 Dec 2010 13:16 UTC

Replying to alec:

Use (g)zip.
it's bzipped

[rcubetrac]

Comment by @alecpl on 8 Dec 2010 13:17 UTC

I hope it contains only your request. Send it to alec at alec.pl.

[rcubetrac]

Comment by altex on 8 Dec 2010 13:19 UTC

Replying to alec:

I hope it contains only your request. Send it to alec at alec.pl.
Only my

[rcubetrac]

Comment by altex on 8 Dec 2010 13:19 UTC

Replying to alec:

Request timeout is 180 sec. (app.js) So, maybe your IMAP server just doesn't finish a response in this time.

Yes, i think so. There are many lines in log after request is finished.
RC trying to fetch 112 000 letters and i don't know why.

[rcubetrac]

Comment by @alecpl on 8 Dec 2010 13:21 UTC

Probably for sorting. Set list sorting to "None". That should help.

[rcubetrac]

Comment by @alecpl on 8 Dec 2010 13:45 UTC

I've seen the log. Gmail and/or your connection is extremly slow. Only solution for you is setting sorting to "None". I'm re-closing this ticket.

[rcubetrac]

Status changed by @alecpl on 8 Dec 2010 13:45 UTC

reopened => closed

[rcubetrac]

Milestone changed by @alecpl on 8 Dec 2010 13:45 UTC

0.5-stable => 0.2-stable

[rcubetrac]

Comment by altex on 8 Dec 2010 18:15 UTC

Thanks, sorting:none helped.

And i tested search "in all mail folder", it works fine.