≑ Menu

6 rsync Examples to Exclude Multiple Files and Directories using exclude-from

Rsync is very powerful tool to take backups, or sync files and directories between two different locations (or servers).

You know this already, as we presented you with practical examples on rsync earlier.

In a typical backup situation, you might want to exclude one or more files (or directories) from the backup. You might also want to exclude a specific file type from rsync.

This article explains how to ignore multiple files and/or directories during rsync with examples.

First, create a sample directory structure as shown below (with some empty files) that can be used for testing purpose.

$ cd ~
$ mkdir -p source/dir1/dir2
$ mkdir -p source/dir3
$ touch source/file1.txt
$ touch source/file2.txt
$ touch source/dir1/dir2/file3.txt
$ touch source/dir3/file4.txt

The above command will create a source directory (under your home directory) with the following structure.

source
- file1.txt
- file2.txt
- dir1
  - dir2
    - file3.txt
- dir3
  - file4.txt

1. Exclude a specific directory

If you don’t want to sync the dir1 (including all it’s subdirectories) from the source to the destination folder, use the rsync –exclude option as shown below.

$ rm -rf destination

$ rsync -avz --exclude 'dir1' source/ destination/
building file list ... done
created directory dest
./
file1.txt
file2.txt
dir3/
dir3/file4.txt

Verify to make sure dir1 is not copied from source directory to destination directory.

$ find destination
destination
destination/file2.txt
destination/file1.txt
destination/dir3
destination/dir3/file4.txt

2. Exclude multiple directories that matches a pattern

The following example will exclude any directory (or subdirectories) under source/ that matches the pattern “dir*”

$ rm -rf destination

$ rsync -avz --exclude 'dir*' source/ destination/
building file list ... done
created directory destination
./
file1.txt
file2.txt

Verify the destination directory to make sure it didn’t copy any directories that has the keyword “dir” in it.

$ find destination
destination
destination/file2.txt
destination/file1.txt

3. Exclude a specific file

To exclude a specific file, use the relative path of the file in the exclude option as shown below.

$ rm -rf destination

$ rsync -avz --exclude 'dir1/dir2/file3.txt' source/ destination/
building file list ... done
created directory destination
./
file1.txt
file2.txt
dir1/
dir1/dir2/
dir3/
dir3/file4.txt

Verify the destination directory to make sure it didn’t copy the specific file ( dir1/dir2/file3.txt in this example).

$ find destination
destination
destination/file2.txt
destination/file1.txt
destination/dir1
destination/dir1/dir2
destination/dir3
destination/dir3/file4.txt

4. Exclude path is always relative

If you are not careful, you might make this mistake.

In the following example, the exclude option seems to have a full path (i.e /dir1/dir2/file3.txt). But, from rsync point of view, exclude path is always relative, and it will be treated as dir1/dir2/file3.txt. In the example below, rsync will look for dir1 under source directory (and not under / root directory).

$ rsync -avz --exclude '/dir1/dir2/file3.txt' source/ destination/

So, the above command is exactly same as the following. Just to avoid confusion (and to make it easy to read), don’t give / in front of the exclude path.

$ rsync -avz --exclude 'dir1/dir2/file3.txt' source/ destination/

5. Exclude a specific file type

To exclude a specific file type that has a specific extension, use the appropriate pattern. For example, to exclude all the files that contains .txt as extension, do the following.

$ rsync -avz --exclude '*.txt' source/ destination/
building file list ... done
created directory destination
./
dir1/
dir1/dir2/
dir3/

Verify the destination directory to make sure it didn’t copy the *.txt files.

$ find destination
destination
destination/dir1
destination/dir1/dir2
destination/dir3

Note: The above is very helpful, when you want to backup your home directory, but exclude all those huge image and video files that has a specific file extension.

6. Exclude multiple files and directories at the same time

When you want to exclude multiple files and directories, you can always specify multiple rsync exclude options in the command line as shown below.

$ rsync -avz --exclude file1.txt --exclude dir3/file4.txt source/ destination/

Wait. What if I had tons of files that I want to exclude from rsync?

I can’t keep adding them in the command line using multiple –exclude, which is hard to read, and hard to re-use the rsync command for later.

So, the better way is to use rsync –exclude-from option as shown below, where you can list all the files (and directories) you want to exclude in a file.

First, create a text file with a list of all the files and directories you don’t want to backup. This is the list of files and directories you want to exclude from the rsync.

$ vim exclude-list.txt
file1.txt
dir3/file4.txt

Next, execute the rsync using –exclude-from option with the exclude-list.txt as shown below.

$ rm -rf destination

$ rsync -avz --exclude-from 'exclude-list.txt' source/ destination/
building file list ... done
created directory destination
./
file2.txt
dir1/
dir1/dir2/
dir1/dir2/file3.txt
dir3/

Verify the desitination directory to make sure the files and directories listed in the exclude-list.txt file is not backed-up.

$ find destination
destination
destination/file2.txt
destination/dir1
destination/dir1/dir2
destination/dir1/dir2/file3.txt
destination/dir3

If you enjoyed this article, you might also like..

  1. 50 Linux Sysadmin Tutorials
  2. 50 Most Frequently Used Linux Commands (With Examples)
  3. Top 25 Best Linux Performance Monitoring and Debugging Tools
  4. Mommy, I found it! – 15 Practical Linux Find Command Examples
  5. Linux 101 Hacks 2nd Edition eBook Linux 101 Hacks Book

Bash 101 Hacks Book Sed and Awk 101 Hacks Book Nagios Core 3 Book Vim 101 Hacks Book

{ 26 comments… add one }

  • ashwin January 25, 2011, 5:03 am

    Thanks for this information, very useful.

    “Small typo error at point 6, I think it should be files and directories (Instead of ‘are’).”

  • carlos January 25, 2011, 8:01 am

    Good post,its very usefull. thank you very much!!! from argentina.

  • Felix Frank January 25, 2011, 10:32 am

    As always, I feel compelled to amend πŸ˜‰

    The leading slash ‘/’ in the exclude is of great importance. It means “match only at the base of the source tree”. Assume the following:
    – tools/
    – src/
    –program.c
    –tools/

    So there is the directory tools/ and src/tools/. –exclude tools/ should exclude both of those directories named “tools”, whereas –exclude /tools/ will only exclude the former (which is probably more often intended).

  • Ramesh Natarajan January 25, 2011, 10:01 pm

    @Ashwin,

    Thanks for pointing out the typo. It is fixed.

    @Felix,

    Your amendment are always most welcome, and they are helpful. Thanks.

  • Jonathan Lumb February 16, 2011, 4:35 pm

    Thank you for taking the time to write such a systematic guide to understanding and using the exclude option in Rsync – very useful and cleared up some points I was uncertain about!

    Jonathan

  • sathiya March 28, 2011, 1:47 am

    It saved my time !!!

    Thanks.

  • Sumesh April 13, 2011, 9:35 am

    Really helpful doc. Thanks for sharing.

  • Draget July 11, 2011, 1:48 pm

    As Felix pointed out, the leading / is of importance. So point 4 should be corrected!

  • sandeep krishnan August 8, 2011, 3:11 am

    Hello Ramesh sir i enjoy reading this article it would be nice if you specify how to use rsync with ssh on different port

  • Noel September 8, 2011, 11:12 am

    Thanks for the post. There is one case you did not specify, which is also very handy. I found from here.

    Excluding all files but a specific match or directory and its subfiles.

    rsync –include=’*/’ –include=’Dir/*’ –include=’*.jpg’ –exclude=’*’ -m

    From what I can tell the ‘*/’ includes all directories – you need all parent directories of desired files included – and the -m then prunes all empty directories that had no matching files.

  • woo September 21, 2011, 5:36 pm

    The “–exclude” option works like a pattern, so:

    rsync -av –delete /home/me/.mozilla –exclude=Cache /media/hddext/backup

    will exclude any folder which has a name of “Cache”.

  • miguelwill October 30, 2011, 7:36 pm

    remember –exclude-from option:

    /tmp/exclude-files:
    – *.mp3
    – *.avi
    – *.iso
    – *.mpg

    rsync -hva –exclude-from=/tmp/exclude-files /source /destination

  • joe March 30, 2012, 9:39 am

    great post.
    thanks :)

  • Wehibe June 27, 2012, 11:09 am

    Thanks for taking the time to post this. It is handy and very helpful
    Cheers

  • Arthur N. Ketcham July 25, 2012, 2:49 pm

    Using the –exclude option, –exclude ‘dir*’ will not work correctly. With globbing, you need to use double quotes. So that should be corrected to:
    –exclude”dir*”

  • JD December 5, 2012, 9:24 pm

    IMPORTANT NOTE:

    NO SPACES AT THE END OF EACH LINE for your –exclude-from file

    AND

    Exclude path is always relative is NOT TRUE exclude path is only relative to the source directory if you begin your patterns with ‘/’ otherwise the pattern will match ANYWHERE in the source directory…

  • iBro December 12, 2012, 8:42 pm

    Yes, I agree with JD.

    I tried with the following structure and I tried to exlude only ‘dir1/dir3′ , but when I use ‘–exclude dir3′, ‘dir1/dir2/dir3′ and ‘dir1/dir4/dir3′ also get excluded.

    /tmp $ mkdir -p dir1/dir{2,3,4}/dir{1,2,3}
    /tmp $ find dir1
    dir1
    dir1/dir3
    dir1/dir3/dir1
    dir1/dir3/dir3
    dir1/dir3/dir2
    dir1/dir2
    dir1/dir2/dir1
    dir1/dir2/dir3
    dir1/dir2/dir2
    dir1/dir4
    dir1/dir4/dir1
    dir1/dir4/dir3
    dir1/dir4/dir2

    /tmp $ rsync -av –exclude dir3 dir1/ dir2/
    sending incremental file list
    ./
    dir2/
    dir2/dir1/
    dir2/dir2/
    dir4/
    dir4/dir1/
    dir4/dir2/

    sent 146 bytes received 39 bytes 370.00 bytes/sec
    total size is 0 speedup is 0.00

    /tmp $ find dir2
    dir2
    dir2/dir2
    dir2/dir2/dir1
    dir2/dir2/dir2
    dir2/dir4
    dir2/dir4/dir1
    dir2/dir4/dir2

  • Owen March 19, 2013, 11:51 pm

    So am i correct in assuming that
    “the excluded files path is relative to the source path iff the source path is absolute”
    ?

  • everge48 March 21, 2013, 1:17 pm

    Here are some examples of exclude/include matching:

    –exclude “*.o” would exclude all filenames matching *.o

    –exclude “/foo” would exclude a file called foo in the transfer-root directory

    –exclude “foo/” would exclude any directory called foo

    –exclude “/foo/*/bar” would exclude any file called bar two levels below a directory called foo in the transfer-root directory

    –exclude “/foo/**/bar” would exclude any file called bar two or more levels below a directory called foo in the transfer-root directory

    –include “*/” –include “*.c” –exclude “*” would include all directories and C source files

    –include “foo/” –include “foo/bar.c” –exclude “*” would include only foo/bar.c (the foo/ directory must be explicitly included or it would be excluded by the “*”)

  • Jordan April 17, 2013, 12:54 pm

    Hi Ramesh…
    One more time your tip was so helpfull.
    Keep writting!
    Tks!
    Jordan

  • Seb May 3, 2013, 7:35 am

    Great stuff, thanks! Always forget the exclude syntax all the common options in one place :)

  • ronak June 6, 2013, 11:23 am

    Hey Guys thnxs for sharing the information.
    I need one help:
    Say i have the directory “log” located at two locations as:
    /a/b/log and
    /a/b/c/log

    i dont want to exclued the first one in “/a/b/log” but only “/a/b/c/log”.

    Thanks

  • Leila October 28, 2014, 12:40 am

    Thank you soooooooooooo much. The instructions were very clear and very helpful!

  • Mark March 27, 2015, 2:33 am

    Hi! You’ve made a mistake in section 4 “Exclude path is always relative”.

    It’s true that it is always relative but it’s NOT the same using the leading slash or not to do so. The leading slash indicates the full qualified path beginning at your given transfer-root directory.

    See also the man page of rsync:

    “””
    Here are some examples of exclude/include matching:
    o “- /foo” would exclude a file (or directory) named foo in the transfer-root directory
    o “- foo/” would exclude any directory named foo

    Kind regards,
    Mark

  • ramy May 23, 2015, 1:15 pm

    very useful post. adding an update- on OS X Yosemite 10.10.3 bash, rsync does not recognize excludes that start with ‘ – e.g. –exclude ‘*.MOV’ is ignored by rsync. Instead replace with ” e.g. –exclude “*.MOV” or –exclude-from “exclude-list.txt”

  • Patrick Goetz June 18, 2015, 4:50 am

    This statement:
    ———————–
    $ rsync -avz –exclude ‘/dir1/dir2/file3.txt’ source/ destination/

    So, the above command is exactly same as the following. Just to avoid confusion (and to make it easy to read), don’t give / in front of the exclude path.

    $ rsync -avz –exclude ‘dir1/dir2/file3.txt’ source/ destination/
    ———————–

    is flat out incorrect. With a slash at the beginning of the exclude value, the pattern is anchored to the top of the directory tree. If you remove the slash then that pattern is matched throughout the entire directory tree (i..e any dir1/dir2/file3.txt further down the tree will be excluded as well).

    Furthermore, it makes a difference if the source ends with a slash or not. Remove the slash from the end of “source/” and the search pattern goes back to the parent directory; i.e.

    $ rsync -avz –exclude ‘/dir1/dir2/file3.txt’ source destination/

    will exclude nothing. Please try and understand how these tools work before writing a blog post like this.

Leave a Comment