[ILUG] rsync over ssh problem

[kevin]

On Wed, 2010-08-18 at 11:25 +0100, Brian O'Mahony wrote:
> My issue is 0 bytes are transferred (just checked) in the first sync at this time. However other things are getting sync'd on the hour. This is just CVS folders. I have a feeling as the folders are so large the reading file lists bit is timing out. However if I do these manually iot works fine, and all the rest (about 15 sync's) work fine after the first fails during the night. Which I find weird.
> 
> I have implemented the change Marcus recommended and will see how it goes tonight.
> 
> B
> 
> -----Original Message-----
> From: ilug-bounces at linux.ie [mailto:ilug-bounces at linux.ie] On Behalf Of kevin
> Sent: Wednesday, August 18, 2010 11:18 AM
> To: Irish Linux Users Group
> Subject: RE: [ILUG] rsync over ssh problem
> 
> On Wed, 2010-08-18 at 11:04 +0100, Brian O'Mahony wrote:
> > FYI I have been having this issue sync'ing data over an MPLS between Dublin & Bangalore on a nightly basis. It only happens on the first data transfer of the night, and things are quiet (ie its 5am BLR time, and 1am Irish time, so there should be no one in the office.)
> > 
> > I would be very interested if you find a solution. To be honest I have ignored this for a good few months (I just scheduled an extra data transfer a minute before all the rest kick off that I know will fail). 
> > B
> > 
> > -----Original Message-----
> > From: ilug-bounces at linux.ie [mailto:ilug-bounces at linux.ie] On Behalf Of kevin
> > Sent: Wednesday, August 18, 2010 10:00 AM
> > To: Irish Linux Users Group
> > Subject: [ILUG] rsync over ssh problem
> > 
> > Part of our backup routine is a cron job that runs rsync over ssh to an
> > offsite computer every night. Recently I have noticed that it fails
> > about 1/2 of the time, not always and a failed rysnc one night might
> > work fine the next night.
> > 
> > The error we get is:
> > 
> > Write failed: Broken pipe
> > rsync: connection unexpectedly closed (13729658 bytes received so far)
> > [receiver]
> > rsync error: error in rsync protocol data stream (code 12) at io.c(601)
> > [receiver=3.0.7]
> > rsync: connection unexpectedly closed (783 bytes received so far)
> > [generator]
> > rsync error: unexplained error (code 255) at io.c(601) [generator=3.0.7]
> > 
> > I have the "KeepAlive Yes" option in the sshd_config file
> > 
> > rsync options used in the batch file are:
> > 
> > rsync -e ssh -avzxHS --numeric-ids --del --delete-excluded
> > --exclude-from "/home/cybercolloids/exclusions.txt" --exclude ".gvfs"
> > 
> > Looking at the log file the only thing that looks odd is that the
> > connection fails when transferring large image files - a few Mb
> > 
> > Any ideas?
> > 
> > Thanks
> > 
> > Kevin.
> > 
> 
> An interesting point - I actually have a batch file that runs through a
> series of rsync commands - one for each partition on the raid array.
> However the first is the /home directory which is the one that changes
> most often and is the most important.
> 
> I will also swap the commands over so that rsync /home is further down
> the list and see if that helps.
> 
> Kevin.
> 
In that case it must be a different problem - my connection starts up
and transfers some files but then breaks part way through. If you are
using rsync over ssh the adding this to your sshd_config file should
help:

KeepAlive Yes
ClientAliveInterval 30