Removing duplicated files in server

kalepond

I have a wordpress install with heaps of duplicated images and file uploads, really, its a mess.
Need to reduce the size of the website and I thought I could run a bash script to remove the duplicated files and replace them with hard links.
Now, would this work? has anyone attempted before?

WSS

sha256sum your files, compare. Obviously check sizes/etc, but a collision is not horribly likely. I'd suggest a soft link.

comi

If you do not need files to be able to diverge later, then yes it would work - relatively straightforward, see above.

Otherwise you need some pro dedup tool (or filesystem).

Mr_Tom

I have used fdupes in the past with some success. You could setup a script/line to remove any, but I generally just have the output put into a text file to manually review before removing anything

sonic

Increase disk space and forget it.

InceptionHosting

I remember going through a similar thing about 10 years ago, it did not end well.

seanho

Another possibility is to copy/rebase onto a btrfs filesystem and use bedup (extent-panel dedup). Then you get copy-on-write if you need to make modifications. ZFS is another option.

Janevski

I disagree.
You need to increase website size.
The bigger the wordpress, the stronger you become.

mywebhosting

I have used fslint and it worked for me.