Using tar and netcat to backup filesystems: A pitfall resulting in broken links and how to avoid it

As shown in yesterday’s blog entry, I failed to recover my Feisty laptop due to missing links. Remembering the old saying “Every fool may do a backup” I now managed to find my mistake.

The error may be shown using localhost, so here we go:

First I started the listening netcat like this:

cd /tmp
netcat -l -p 2342 | tar xvf -

Then I backuped /sbin using:

cd /
tar cf - sbin | netcat localhost 2342

Both processes do not stop, until you hit CTRL-C. And at this point I did a mistake: I hit CTRL-C on the receiving side (I saw no further files coming in). DO NOT DO THIS – IT WILL EVENTUALLY LEAVE YOU WITH BROKEN LINKS. E.g. instead of

root@lulu:/tmp# ls -l sbin/ip
lrwxrwxrwx 1 root root 7 2008-01-16 11:33 sbin/ip -> /bin/ip

you will have

root@lulu:/tmp# ls -l sbin/ip
---------- 1 root root 0 2008-01-16 11:28 sbin/ip

Let us dig a little bit deeper using strace. Again start the listening side and then the sending side. Do not hit CTRL-C. Looking at the process list you will see that the sending tar has finished, the receiving tar is still there. Strace this process and hit CTRL-C on the sending side. Check sbin/ip and you will find a link correctly pointing to /bin/ip.

This is the strace output:

root@lulu:/tmp# cat receiving-tar.strace
8394 read(0, "", 10240) = 0
8394 clock_gettime(CLOCK_REALTIME, {1200479622, 923188961}) = 0
8394 clock_gettime(CLOCK_REALTIME, {1200479622, 923263552}) = 0
8394 close(0) = 0
8394 SYS_299(0xffffff9c, 0x808e6a5, 0xbfd650fc, 0x808e6a5, 0xb7f6eff4) = 0
8394 chmod("sbin", 0755) = 0
8394 chown32("sbin", 0, 0) = 0
8394 lstat64("sbin/lsmod", {st_mode=S_IFREG, st_size=0, ...}) = 0
8394 unlink("sbin/lsmod") = 0
8394 symlink("/bin/lsmod", "sbin/lsmod") = 0
8394 lchown32("sbin/lsmod", 0, 0) = 0
8394 lstat64("sbin/ip", {st_mode=S_IFREG, st_size=0, ...}) = 0
8394 unlink("sbin/ip") = 0
8394 symlink("/bin/ip", "sbin/ip") = 0
8394 lchown32("sbin/ip", 0, 0) = 0
8394 close(1) = 0
8394 munmap(0xb7cff000, 4096) = 0
8394 exit_group(0) = ?

Obviously tar handles the symbolic links not in the moment as they appear but in some final procedure. When I hitted CTRL-C on the receiving side I prevented tar to run that final procedure as this strace clearly shows:

root@lulu:/tmp# cat receiving-tar-ctrl-c
8495 read(0, "", 10240) = 0
8495 --- SIGINT (Interrupt) @ 0 (0) ---

So you really must hit CTRL-C only on the sending side, interrupting netcat, which is safe as the sending tar already has gone.

Even better: Use netcat’s q option for the sending netcat, e.g.

Receiving process:

root@lulu:/tmp# netcat -l -p 2342 | tar xvf -

Sending side:

root@lulu:/# tar cf - sbin | netcat -q 2 localhost 2342

With -q 2, netcat stop go away 2 secs after detecting EOF on stdin. The receiving netcat then will go away too.

So my conclusion:

Always use tar + netcat this way:

Receiving side:

netcat -l -p 2342 | tar xvf –

Sending side:

tar cf – whatever-you-want-to-backup | netcat -q 2 localhost 2342

Never omit -q !


Just a hint for those using e.g. SuSE: Though netcat version is 1.10 (same as in Feisty) option -q ist not available (there are others missing too). So in SuSE you must hit CTRL-C on the right side …


Tags: , , , , ,

8 Responses to “Using tar and netcat to backup filesystems: A pitfall resulting in broken links and how to avoid it”

  1. Anonymous Says:

    free car quote

    Excellent post. Keep it up!

  2. Adam Monsen Says:

    The -q flag works well, thanks! I also usually add -v.

  3. szrwejyti sujchk Says:

    vifm uszxetmiy udlfytaro pkjhwb lsedipo rsynqmhed abtcvs

  4. Alexwebmaster Says:

    Hello webmaster
    I would like to share with you a link to your site
    write me here

  5. How to Get Six Pack Fast Says:

    My fellow on Orkut shared this link and I’m not dissapointed at all that I came here.

  6. Khalil Says:

    Cool ! Thanks ! Nice way to ‘stream’ a backup , if you have no space for the temporary image !

  7. mbv Says:

    Or just use -w for timeout:
    $ tar cf – sbin | netcat -w 2 localhost 2342

    This will cause netcat to end both sides after the timeout (it’s actually in the README!)

  8. Kurtis Seilhamer Says:

    What i don’t realize is in fact how you’re no longer actually a lot more neatly-liked than you may be right now. You are very intelligent. You understand therefore significantly in terms of this subject, made me in my view imagine it from so many varied angles. Its like men and women are not interested until it is one thing to do with Woman gaga! Your personal stuffs outstanding. All the time maintain it up!

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s

%d bloggers like this: