≡ Menu

How to delete a large number of files on Linux?

I had about 6 million files that had to be deleted on a Linux based system (CentOS). I needed a solution that works and is fast. The Linux command rm would probably take hours to delete 6 million files. In addition to that, rm simply cannot delete so many files and it errors out with a message argument list too long. 

To get around this, I found out that there is a command in Linux called find. This command also takes as an argument the file names to find and what to do with them. Note that find is a recursive command, so it will act on all files in a directory and any sub-directories within the current directory. So make sure that you are in the right directory before using this command. As in all Linux based systems, there is no recycle bin, so use the following command with caution.

The command find can take as an argument a -delete argument which instructs find to delete the files that it finds.

For example:

find . -delete

This will delete all files in a directory and all files in sub-directories. It will also delete the sub-directories within the current directory.

If you want to delete files with a specific name or extension, you can use the -name argument.

For example:

find . -name *.csv -delete

This command will delete all CSV files in a directory and all of its sub-directories.

If you want to delete all files, but keep all the directories, the following command is useful:

find . -type f -delete

I cannot stress enough that find works on all directories and sub-directories, and once a file is deleted, it cannot be recovered.

Another thing about the find command is that it is very fast, as compared to rm. Deleting a million files took about a minute for find which would have taken an hour for rm, even if the argument list was not long enough for rm to choke.

You can also use the rm command along with find. The following command uses rm as an argument to find to delete files.

find . -exec rm {} \;

This command instructs find to execute rm for each file it finds. The {} argument to rm is the file name. This is equivalent to using the argument -delete to find, though I don’t know how fast this will be.

So if you need a fast way to delete files on a Linux machine, I presented to you a few options that could be used. Hope this will be useful to you!

Rahul

{ 0 comments… add one }

Leave a Comment