Quote:
Note: I think it has something to do with Windows memory swapping/caching, but I have no idea how to do anything about that.
Less to do with Windows, more to do with hardware.
Having become a performance expert over the last year or so, I strongly suspect your problem is that reading from memory takes time.
The more memory you have in use, the lower the percentage of all the data read will be in the CPU caches, and therefore it'll be hitting the RAM more frequently.
I had a similar problem. Having done lots of profiling and tuning, I was eventually in the situation where 25% of my execution time was being spent on one simple line of code.... I eventually discovered that this was the point that the referenced object was loaded from memory, and more often that not, that object wasn't in the CPU cache.
So the good news is, there's your answer - the bad news is that doing anything about it is very very tricky.
Edit:
As an extension to that - Arrays are good as they enforce some kind of locality of memory between objects. If you keep objects that are frequently referenced in quick succession next to each other in an array this will speed things up.
When the CPU requests something from RAM, it doesn't just grab what the CPU asks for, but will grab a page of memory and put all of that in the CPU cache. Therefore if you're using an array it'll sweep a whole bunch of objects into the CPU cache, and ideally you want those objects to be useful rather that a load of stuff you couldn't care less about. So yes, arrays of data with objects accessed together adjacent to each other is the way forwards.