Double trouble

• Article 23 of 26
• M-iD, September 2005

Page 1 | Page 2 | Page 3 | All 3 Pages

Sooner or later, however, duplicates will arise and organisations need to consider how best to deal with them. An automated approach with some manual intervention is the best option for most organisations. Some of the newer content management systems have de-duplication functions built in, and there are products such as Mobius's ViewDirect-ABS that claim to reduce duplication as well. In older systems, reporting tools that sort documents by metadata may be able to produce an inventory of similar documents.

Iron Mountain's Maurice advises his clients to set up the inventory of master documents first, using metadata attributes such as creator, creation date, modification date and so on to identify the master documents. They can then be sure which is the duplicate or the redundant version and which is the document they should retain.

Grant Edgar, managing consultant at Open Text Global Services, says that most organisations should use the identification of duplicates as a chance to understand where processes are going wrong, rather than try to fix the problem retroactively unless it is a significant storage problem or an impediment to navigation. "They can then change the process and instruct the staff accordingly," he says.

If two duplicate documents are both declared records, organisations need to be very careful before deleting them. If the two documents are identical, then creating a hash of one of the documents and storing that along with an audit log of the deletion should be enough to convince an auditor that the organisation deleted the record properly. For absolute peace of mind, the record metadata of the deleted document might be kept with a pointer indicating the remaining document contains the file content.

However, once documents start to differ in metadata or content, it becomes far harder to argue convincingly that they and their record data needed to be deleted. Open Text's Caughell says that the key thing auditors look for is consistency in how organisations conduct processes, and provided that consistency is maintained, deletion of 'fuzzy' duplicates may be possible.

However, Mike Blake, of IBM's information management software division, says that by the time a duplicate has been declared a record, it's too late to delete it. "In records management, you really are trying to remove the power of deletion except through policy. Once something's declared a record, it's sacrosanct, it's in the system and it's not going away."