Tweet
Many of the internal algorithms were rewritten in V5 to use less RAM and a lot of code profiling was done to find sections of the code that were overly CPU intensive. We then rewrote those sections to be more frugal with the CPU.
The result is an overall reduction in CPU load, RAM usage and indexing time. This was done at the same time as we added many new features. Quite an achievement to add so many new features and use less resources than before.
This means Zoom is better able to be run as a background task and won't total hog all the resources of the PC running the indexer. This means you can continue to use your PC or server while Zoom is running.
Here are a few benchmark graphs that compare the indexing process of V4 of Zoom to V5 (beta build 11).
Scenario:
PC: Dual Xeon CPU 3.4 Ghz, 1GB RAM
Net connection: ADSL 1.5Mb/sec
Settings: Spider mode, 2 threads, CRC on, No caching
Site indexed: Remote site, high latency (360ms ping), mostly dynamic pages from a database.
Results:
V5 used significantly less CPU time (86 seconds) compared to V4 (108 second) for the indexing of the 1300 pages. In both V4 and V5 the CPU load worked out to be roughly 10%, but as V5 finish processing earlier the overall load was less. The CPU load % was a low 10% in this case partially because we had dual CPUs sharing the load and partially because there was such high latency on the remote site. A lot of time was spent waiting for the remote server to respond.
This graph shows the peak virtual memory usage of the two processes. Although 19% is a nice improvement, the improvements will be much more dramatic as the number of pages indexed grows to beyond 250,000 files. Reductions in RAM usage of 50% or more, are to be expected in that case.
Better algorithms means faster processing, which means less time indexing.
The result is an overall reduction in CPU load, RAM usage and indexing time. This was done at the same time as we added many new features. Quite an achievement to add so many new features and use less resources than before.
This means Zoom is better able to be run as a background task and won't total hog all the resources of the PC running the indexer. This means you can continue to use your PC or server while Zoom is running.
Here are a few benchmark graphs that compare the indexing process of V4 of Zoom to V5 (beta build 11).
Scenario:
PC: Dual Xeon CPU 3.4 Ghz, 1GB RAM
Net connection: ADSL 1.5Mb/sec
Settings: Spider mode, 2 threads, CRC on, No caching
Site indexed: Remote site, high latency (360ms ping), mostly dynamic pages from a database.
Results:
V5 used significantly less CPU time (86 seconds) compared to V4 (108 second) for the indexing of the 1300 pages. In both V4 and V5 the CPU load worked out to be roughly 10%, but as V5 finish processing earlier the overall load was less. The CPU load % was a low 10% in this case partially because we had dual CPUs sharing the load and partially because there was such high latency on the remote site. A lot of time was spent waiting for the remote server to respond.
This graph shows the peak virtual memory usage of the two processes. Although 19% is a nice improvement, the improvements will be much more dramatic as the number of pages indexed grows to beyond 250,000 files. Reductions in RAM usage of 50% or more, are to be expected in that case.
Better algorithms means faster processing, which means less time indexing.