Wednesday, December 17, 2014

Transferring folder with rsync - example of a content sync vs complete folder transer

Make test folder with 10 empty files:

mkdir test
touch test/test{1..10}

Use rsync to transfer test folder to a server (options: -a archive, similar to -r and means sync recursively and preserve symbolic links, modification times, permissions, -v verbose, -z reduce network transfer by adding compression, -P combines flags --progress and --partial, first gives you progress bar for the transfers and second will allow resuming of the interrupted transfers :

rsync -chavzP --stats test/ mpjanic@zoran.stanford.edu:~/test/
mpjanic@zoran.stanford.edu's password: 
building file list ... 
11 files to consider
created directory /home/mpjanic/test
./
test1
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=9/11)
test10
           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=8/11)
test2
           0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=7/11)
test3
           0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=6/11)
test4
           0 100%    0.00kB/s    0:00:00 (xfer#5, to-check=5/11)
test5
           0 100%    0.00kB/s    0:00:00 (xfer#6, to-check=4/11)
test6
           0 100%    0.00kB/s    0:00:00 (xfer#7, to-check=3/11)
test7
           0 100%    0.00kB/s    0:00:00 (xfer#8, to-check=2/11)
test8
           0 100%    0.00kB/s    0:00:00 (xfer#9, to-check=1/11)
test9
           0 100%    0.00kB/s    0:00:00 (xfer#10, to-check=0/11)

Number of files: 11
Number of files transferred: 10
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 305
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 717
Total bytes received: 246

sent 717 bytes  received 246 bytes  214.00 bytes/sec
total size is 0  speedup is 0.00

Previous command created test folder on the server and sync the contents with the test folder on our computer. Note that specifying the contents of the folder was done with a slash sign by specifying test/ and ~/test/ 
The same could be done with the command without the slash sign in which case the whole folder will be transferred, and not the contents of the folder, so in the second part of the rsync command we have to specify only the home folder of the server where the test folder will be transferred to, i.e. ~

rsync -chavzP --stats test mpjanic@zoran.stanford.edu:~
mpjanic@zoran.stanford.edu's password: 
building file list ... 
11 files to consider
test/
test/test1
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=9/11)
test/test10
           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=8/11)
test/test2
           0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=7/11)
test/test3
           0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=6/11)
test/test4
           0 100%    0.00kB/s    0:00:00 (xfer#5, to-check=5/11)
test/test5
           0 100%    0.00kB/s    0:00:00 (xfer#6, to-check=4/11)
test/test6
           0 100%    0.00kB/s    0:00:00 (xfer#7, to-check=3/11)
test/test7
           0 100%    0.00kB/s    0:00:00 (xfer#8, to-check=2/11)
test/test8
           0 100%    0.00kB/s    0:00:00 (xfer#9, to-check=1/11)
test/test9
           0 100%    0.00kB/s    0:00:00 (xfer#10, to-check=0/11)

Number of files: 11
Number of files transferred: 10
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 310
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 722
Total bytes received: 246

sent 722 bytes  received 246 bytes  176.00 bytes/sec
total size is 0  speedup is 0.00

In case you delete first 5 files on the server 

rm test{1..5}

With the first command you will transfer only the files missing:

rsync -chavzP --stats test/ mpjanic@zoran.stanford.edu:~/test/
mpjanic@zoran.stanford.edu's password: 
building file list ... 
11 files to consider
./
test1
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=9/11)
test2
           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=7/11)
test3
           0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=6/11)
test4
           0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=5/11)
test5
           0 100%    0.00kB/s    0:00:00 (xfer#5, to-check=4/11)

Number of files: 11
Number of files transferred: 5
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 305
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 522
Total bytes received: 136

sent 522 bytes  received 136 bytes  188.00 bytes/sec
total size is 0  speedup is 0.00

Also the second command will recognize that the folder it is transferring exist and will transfer only 5 files missing:

rsync -chavzP --stats test mpjanic@zoran.stanford.edu:~
mpjanic@zoran.stanford.edu's password: 
building file list ... 
11 files to consider
test/
test/test1
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=9/11)
test/test2
           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=7/11)
test/test3
           0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=6/11)
test/test4
           0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=5/11)
test/test5
           0 100%    0.00kB/s    0:00:00 (xfer#5, to-check=4/11)

Number of files: 11
Number of files transferred: 5
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 310
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 527
Total bytes received: 136

sent 527 bytes  received 136 bytes  147.33 bytes/sec
total size is 0  speedup is 0.00

However, if you want to use first example i.e. syncing of the folder contents and do not type slash after folder name in the command line, rsync will consider complete folder to be transferred and it will transfer complete test folder to the test folder specified for the server. Try to avoid this error.

rsync -chavzP --stats test mpjanic@zoran.stanford.edu:~/test/
mpjanic@zoran.stanford.edu's password: 
building file list ... 
11 files to consider
test/
test/test1
           0 100%    0.00kB/s    0:00:00 (xfer#1, to-check=9/11)
test/test10
           0 100%    0.00kB/s    0:00:00 (xfer#2, to-check=8/11)
test/test2
           0 100%    0.00kB/s    0:00:00 (xfer#3, to-check=7/11)
test/test3
           0 100%    0.00kB/s    0:00:00 (xfer#4, to-check=6/11)
test/test4
           0 100%    0.00kB/s    0:00:00 (xfer#5, to-check=5/11)
test/test5
           0 100%    0.00kB/s    0:00:00 (xfer#6, to-check=4/11)
test/test6
           0 100%    0.00kB/s    0:00:00 (xfer#7, to-check=3/11)
test/test7
           0 100%    0.00kB/s    0:00:00 (xfer#8, to-check=2/11)
test/test8
           0 100%    0.00kB/s    0:00:00 (xfer#9, to-check=1/11)
test/test9
           0 100%    0.00kB/s    0:00:00 (xfer#10, to-check=0/11)

Number of files: 11
Number of files transferred: 10
Total file size: 0 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 310
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 722
Total bytes received: 246

sent 722 bytes  received 246 bytes  276.57 bytes/sec
total size is 0  speedup is 0.00

Now the contents of the test folder on the server are test6-10 files plus the complete test folder transferred:

mpjanic@zoran:~/test$ ls -l
total 4
drwxr-xr-x 2 mpjanic mpjanic 4096 Dec 17 14:51 test
-rw-r--r-- 1 mpjanic mpjanic    0 Dec 17 14:51 test10
-rw-r--r-- 1 mpjanic mpjanic    0 Dec 17 14:51 test6
-rw-r--r-- 1 mpjanic mpjanic    0 Dec 17 14:51 test7
-rw-r--r-- 1 mpjanic mpjanic    0 Dec 17 14:51 test8
-rw-r--r-- 1 mpjanic mpjanic    0 Dec 17 14:51 test9

No comments:

Post a Comment