[ILUG] ftp.heanet.ie

[Paul Jakma]

On Sun, 11 Jan 2004, Colm MacCarthaigh wrote:

> We've looked at Opteron, but it's not available on the
> infrastructure we need 

And it doesnt /appear/ to have the IO infrastructure needed for 64bit 
machines either.

> - itanium is there, but it's simply too costly for what we're doing
> right now 

Curious, is it as expensive as sparc64 and alpha? more expensive
even? :)

> service in its own right.  So there's a lot of factors that lead to
> us keeping it on x86 for now :)

aha :)

> ftp.heanet.ie has been in operation now for about 16 months, in
> that time we've spent about one 10th of the annual budget of our
> European equivalents. We outperform them all, by a factor of at
> least 10 and in most cases more. They all run on Sparc64, and all
> on multiple Sparc64 boxes at that.

wowser. well done.

though still, i bet you could outperform them even more on good 64bit 
hardware. :)

> I remember a few months ago someone suggested we look at HP kit,
> since we run alphas in production and since ftp.kernel.org runs on
> one - it must be good. Then the people at kernel.org added that
> "Current utilisation" to http://www.kernel.org/, and it turns out
> we're busier than ftp.kernel.org - usually by a factor of 2. After
> that, comparisons seemed hardly worthwhile!

wow.

> We run the numbers, and tweak :) That extra 8Gb of RAM isnt for
> direct IO exactly, it's for caching so that we never have to hit
> disk - that's it's major function.

Ok, that makes sense. What kind of 'hit ratio' do you get on that 
8GB<->6TB of cache? Presumably a subset of that 6TB is more popular 
than the rest.

> The rest is there because even with PAE, it's still *much* better
> to cache the files in VM than to pull it from disk. So at any given
> time, we should be able to serve most requests with memory
> operations, rather than disk. 

Neat. Actually, apache and rsync use sendfile() dont they? I wonder 
if the kernel is clever enough to be able to avoid a bounce-buffer 
for sendfile? (though, still your 8GB above 4GB has to be 
bounce-buffered).

> fast disks and 64bit scalable architechtures, we have 6Tb of IDE
> disks at the back of it all, with x86 and clever caching in the
> middle - but it turns out to be a lot better value for money than
> alternatives :)

Yes. Value for money is something the 64bit platform vendors dont 
seem to appreciate. Though, how much would an IBM iSeries ppc64 box 
be? Are they any cheaper than HP or Sun? (And IBM actually support 
Linux on ppc64).

> In it's current configuration, ftp.heanet.ie has peaked at
> 444Mbit/sec, and our current requests per second threshold is over
> 1,300. In testing, we've happily saturated it's Gigabit ethernet
> without complaint.

I still bet ya though you could improve greatly on that 444Mbit/s
with decent 64bit hardware. Whether its worth the cost of the
inflated prices... :)

Anyway, very neat :)

regards,
-- 
Paul Jakma	paul at clubi.ie	paul at jakma.org	Key ID: 64A2FF6A
	warning: do not ever send email to spam at dishone.st
Fortune:
Real computer scientists don't program in assembler.  They don't write
in anything less portable than a number two pencil.