Tweaking Performance - Magic Moments

Every once in while a developer experiences magic moments. Yesterday I had one. A colleague of mine and I were struggling to improve the performance of a data exporting module of the system. First we tried to measure how bad it is, to give us a baseline. With small amount of data the export took 5 minutes, but with 10 times larger (real world volume) it took forever. On the first attempt after 14 hours of running it had not yet finished and we had to stop it. On the second attempt (on a more powerful machine) it exited within a few hours with OutOfMemoryException. So, our first priority was to reduce the memory demand of the application.

Initially we started with memory profiling, trying out several commercial profilers. Unfortunately we didn't achieve any clarity (I wonder whose fault is it). So we decided first to think about the code for a while, to analyze what was going on. We refactored parts of it to understand it better and see the big picture. While doing so, we spotted and fixed a few little annoyances, that caught my eye, and then we concentrated on the big issues. We tried to reach a common understanding about the bottlenecks and the ways to remove them, but I think we got caught in analysis paralysis. None of the ideas seemed worthwhile and the more crazy ideas needed big codebase rework. So in order to reach better clarity, I proposed to get away from it for awhile, and to check how the little fixes we did, affected the performance. My guess was that we had made minor memory and speed improvements. When we executed the small volume data test, it took not 5 minutes, but 20 seconds. Then we tried the real-world test and instead of running for many hours (14+) it completed in just 3 minutes. We stared in amazement and wondered whether we could start celebrating. It seemed too good to be true.

Today, first thing in the morning, I started a performance profiler on the old version to find out what was slowing it down (I know, I should have started from there). I found that the 90% of the processing time went for concatenating numerous big strings. This was done the wrong way (i.e. s1 += s2).  Switching to StringBuilder made a huge difference, but we could actually do without it: the strings just had to be written in a file and there was no need at all to keep their concatenation in memory. There were a few more things we did to reduce memory, but still trying data that is 3 times the size of the expected real-world volumes gives an OutOfMemoryException. Let's see if we could tweak it more.