Hi Rachel

You need to develop a business case for dealing with duplicates. What's the cost of deleting them, what's the worst that could happen if you keep them?

Ideally there is only a single source of truth - i.e.: one and only one official record. At the very least you need to ensure that the official "copy" is managed according to your firm's policies and any relevant regulatory requirements.

If the official copy is destroyed per policy and there are duplicates in the organization post-destruction, those duplicates are discoverable, but what is the chance of litigation and what risk is associated with having the duplicates lying around?

In a perfect world I would eliminate the duplicates. However, we don't live or work in a perfect world so you're going to have to do some analysis on how to handle your duplicates.

Hope this helps at least a little.

Cheers!
Chris

Preserving all documents for ever is over expensive and flooding your server with useless documents. Only one copy of orginals and one back up if required should be preserved. You could formulate a retention policy for your document management that will guide you could retain or destroy your documents. It will save space, reduce the overhead and make your system more efficacious. A document destroyed in pursuance of a valid retention policy is less dangerous than a document cdestroyed without a retention policy. Retention policy harbors destruction of documents in due course and defends if any spoliation issue arises in any investigation, audit or litigation. I think the following academic article could help you in understanting the concept of formulating a retention policy but it is not a legal advice.
http://sites.google.com/site/harjit1234/formulatingadocumentretentionpolicy

I hope it will help you to some extent.

My approach would be to identify the master copies and get rid of all duplicates before migrating content to the new system. There are software tools that can help automate identification of duplicates even if the file names are different. It is also labour-intensive (as you've found out) but it's worth spending money upfront to get rid of the "noise". If you don't you'll spend much more in the next phase....

That said,a cost/benefit analysis should be taken to determine effort vs. value vs. risk etc., as Chris has pointed out above.