[ILUG] slow ssh compression performance

[Justin Mason]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1


P at draigBrady.com writes:
> Paul Mc Auley wrote:
> > On Tue, 11 Oct 2005 11:25:31 +0100  P at draigBrady.com wrote:
> > 
> > | I created a 400MB test file on the server full of NUL chars
> > | which should compress really well, and I've confirmed this
> > | by using gzip directly to stream the file:
> > 
> > | $ time ssh root at server "gzip -c /test" > test
> > | real    0m7.797s
> > 
> > What about when you pipe it through gzip -dc at the local end?
> > It's worth remembering there are two machines involved in this transaction.
> > Also what's the local disk performance like?
> 
> Sure, this is a better comparison to the other tests:
> 
> time ssh root at server "gzip -c //test" | gzip -dc > test
> real    0m17.001s
> 
> The client is a 2.8GHz P4 with a fast disk
> 
> > | Not using compression gives OK performance (7MB/s):
> > 
> > | $ time scp admin at server:/test .
> > | real    0m57.286s
> > 
> > | However using compression makes things much worse:
> > 
> > | $ time scp -C admin at server:/test .
> > | real    2m6.191s
> > | $ time scp -C -o CompressionLevel=9 admin at server:/test .
> > | real    1m55.567s
> > | $ time scp -C -o CompressionLevel=1 admin at server:/test .
> > | real    2m3.932s
> > 
> > | Now I know it's probably CPU bound and so compressing
> > | will only slow things down, but not as much as above surely?
> > 
> > I find myself wondering if scp is copying the entire file into memory. How 
> > much memory do you have locally? Does the situation improve if you do this 
> > for a file of 100MB?
> 
> Nah, I've Gigs both sides and there would be no way scp would or
> could do that. For completeness:
> 
> $ time scp root at server:/100MB .
> real    0m16.247s
> $ time scp -C root at server:/100MB .
> real    0m31.125s

Fun!

Well, with both the disk and the network out of the equation entirely,
things still look very rough here:

time ssh jm at localhost dd if=/dev/zero bs=100k count=1000 \| gzip -c | gzip -dc > /dev/null
1000+0 records in
1000+0 records out
102400000 bytes transferred in 3.408006 seconds (30046895 bytes/sec)

real    0m3.605s
user    0m0.838s
sys     0m0.009s

: jm 341...; time ssh -C jm at localhost dd if=/dev/zero bs=100k count=1000 > /dev/null
1000+0 records in
1000+0 records out
102400000 bytes transferred in 34.926818 seconds (2931845 bytes/sec)

real    0m35.201s
user    0m2.462s
sys     0m0.054s


it certainly does appear to be 10 times slower.

Perhaps gzip is just a hell of a lot more effectively optimised for the
processor, than libz is... (running on a Pentium M here btw.)

- --j.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.1 (GNU/Linux)
Comment: Exmh CVS

iD8DBQFDS+ywMJF5cimLx9ARAkwSAKCuAREpxMlG9jrzHrCxq0fFj7krLQCbBngD
FS21aVXUfhX36T3vXu5H/FU=
=GSm1
-----END PGP SIGNATURE-----