1. 

   adamhaeder, Member

   I've seen a few forum posts about exporting reports taking a long time, but they were years old. I'm running the latest version and I'm seeing this behavior as well.

   I have a few reports that return between 20,000 and 25,000 rows of data. The report itself renders in a minute or 2 (depending upon the speed of the client cpu), but when I try to export the report (either xls or csv) I run into memory limits on my web server. I have allocated 1.5GB in php.ini, but that is apparently not enough. Am I just out of luck trying to export data sets this large?

   Thanks
2. 

   myDBR Team, Key Master

   Loading very large amounts of data both rendering and export may take some time. Few considerations about the subject:

   - You should consider turning off sorting as it is done in JavaScript. 20,000 rows is so much that user cannot reasonable go through all that data in screen anyways
   - As the result table column widths are dynamic by default, this will take majority of time from the browser rendering

   Exporting speed and memory usage depends on the export type. The most complex being the native Excel format which is very time and memory consuming. Exporting to XML for example, takes considerably less resources.

   --
   myDBR Team
3. 

   adamhaeder, Member

   That's the strange thing - the html rendering only takes 2 or 3 minutes. It's a load on the client cpu, but that's ok. But the csv export seems to take orders of magnitude longer. I allocated 1.5G to my php process, ran the report and selected Export -> CSV. I was running 'top' on the server and I watched the php process consume more and more memory. After 10 minutes, it finally hit the memory limit and the process was killed by the server. Why is that the html rendering only takes a few minutes but the csv rendering wasn't done after 10 minutes?

   And I know I can't expect my users to usefully go through 20,000 records in a web page, that's why the export is so important - they are running this report in order to get data that they then import into another system. So the html display is of little use to me, I really just need the export to work.
4. 

   myDBR Team, Key Master

   For now the CSV-export is using the same codebase as the Excel one, so that is why it is slow. We should be able to improve the CSV-export speed considerably. We'll take a look at it.

   --
   myDBR Team
5. 

   adamhaeder, Member

   That's great, thank you for taking a look. I don't know how you're doing your csv creation, but I've had really good luck with this open source php csv library: http://code.google.com/p/php-csv-utils/
6. 

   myDBR Team, Key Master

   Adam,
   the newest build contains fully rewritten, much faster and less memory consuming CSV export. Give it a try.

   --
   myDBR Team
7. 

   adamhaeder, Member

   That's great. I just upgraded to the latest build and ran a report that contains ~ 22000 records. It took my browser about 4 minutes to render the page and the csv export took about 3 minutes. Good job! Thanks for addressing this so quickly.
8. 

   myDBR Team, Key Master

   You do know that you can set the report output to be CSV by default, so there is no need to load the page first in HTML?

   --
   myDBR Team

Reply

You must log in to post.