paul at clubi.ie
Sun Jan 11 22:11:35 GMT 2004
On Sun, 11 Jan 2004, Colm MacCarthaigh wrote:
> We've looked at Opteron, but it's not available on the
> infrastructure we need
And it doesnt /appear/ to have the IO infrastructure needed for 64bit
> - itanium is there, but it's simply too costly for what we're doing
> right now
Curious, is it as expensive as sparc64 and alpha? more expensive
> service in its own right. So there's a lot of factors that lead to
> us keeping it on x86 for now :)
> ftp.heanet.ie has been in operation now for about 16 months, in
> that time we've spent about one 10th of the annual budget of our
> European equivalents. We outperform them all, by a factor of at
> least 10 and in most cases more. They all run on Sparc64, and all
> on multiple Sparc64 boxes at that.
wowser. well done.
though still, i bet you could outperform them even more on good 64bit
> I remember a few months ago someone suggested we look at HP kit,
> since we run alphas in production and since ftp.kernel.org runs on
> one - it must be good. Then the people at kernel.org added that
> "Current utilisation" to http://www.kernel.org/, and it turns out
> we're busier than ftp.kernel.org - usually by a factor of 2. After
> that, comparisons seemed hardly worthwhile!
> We run the numbers, and tweak :) That extra 8Gb of RAM isnt for
> direct IO exactly, it's for caching so that we never have to hit
> disk - that's it's major function.
Ok, that makes sense. What kind of 'hit ratio' do you get on that
8GB<->6TB of cache? Presumably a subset of that 6TB is more popular
than the rest.
> The rest is there because even with PAE, it's still *much* better
> to cache the files in VM than to pull it from disk. So at any given
> time, we should be able to serve most requests with memory
> operations, rather than disk.
Neat. Actually, apache and rsync use sendfile() dont they? I wonder
if the kernel is clever enough to be able to avoid a bounce-buffer
for sendfile? (though, still your 8GB above 4GB has to be
> fast disks and 64bit scalable architechtures, we have 6Tb of IDE
> disks at the back of it all, with x86 and clever caching in the
> middle - but it turns out to be a lot better value for money than
> alternatives :)
Yes. Value for money is something the 64bit platform vendors dont
seem to appreciate. Though, how much would an IBM iSeries ppc64 box
be? Are they any cheaper than HP or Sun? (And IBM actually support
Linux on ppc64).
> In it's current configuration, ftp.heanet.ie has peaked at
> 444Mbit/sec, and our current requests per second threshold is over
> 1,300. In testing, we've happily saturated it's Gigabit ethernet
> without complaint.
I still bet ya though you could improve greatly on that 444Mbit/s
with decent 64bit hardware. Whether its worth the cost of the
inflated prices... :)
Anyway, very neat :)
Paul Jakma paul at clubi.ie paul at jakma.org Key ID: 64A2FF6A
warning: do not ever send email to spam at dishone.st
Real computer scientists don't program in assembler. They don't write
in anything less portable than a number two pencil.
More information about the ILUG