Email Archiving ? All or Forever

Current Rating:
(0 ratings)

I have been in conversations with an email archiving company on behalf of a client and feel the need to rant a little.  Hopefully it’s a fairly constructive rant.

Email Archiving Solutions (I contend they are simply tools/systems, not solutions, but hey, maybe that’s just me …) claim to “solve” the compliance and ediscovery woes of organizations with regard to email.  And how do they “solve” compliance and ediscovery woes?  They keep ALL email that is sitting on your mail server.  And they keep it forever, unless you do something proactive to delete it.  Hmmm, so how about being compliant with your own internal policies, like your records management policy and retention schedule?

Clearly keeping all email in order to support search and retrieval for compliance needs and/or for discovery is helpful, but there is absolutely no reason to keep ALL email (in perpetuity, or not); the value is in keeping those emails that NEED to be kept for specific reasons (as defined in your records management policy and retention schedule), THEN being able to search and produce it when necessary, and THEN getting rid of it when it's appropriate.

Any organization who employs a records management policy and records retention schedule, and who employs an email archiving tool, but who DOESN’T actually dispose of email is breaking their own rules, and to the ultimate detriment of the business itself.

So What is an Email Archive Good For?

Email, as we all know, is popular.  We all get far too much of it, and probably send too much too.  The sheer volume of email being sent, received – and kept – in organizations puts a very heavy weight on the email servers; so much so that those servers started to get wobbly and crash.  So, enter Email Archiving Solutions, stage right.

One of the main reasons email archive tools were dreamt up was to alleviate the weight on the servers – move that email that everyone gets and sends – and keeps – onto some other platform for storage so the email server can get back to doing what it does best … receiving and delivering email.  Rather than make people manage email, let’s just move it! 

In a nutshell, it was a way for the IT folks (bless their hearts – I highly doubt it was malicious, just expedient) to ensure stability of mail servers without having to actually manage the information in the first place (just ask most RMs who’s organization has an email archiving tool in place whether or not they were consulted or involved in the selection and/or implementation of it … you’ll likely find the answer is a resounding “no”, followed by a frown and then gritting teeth.)

Manage First, "Archive" Second

Honestly, I’m cool with email archiving systems, and do think they have a place in *some* organizations.  Hey, I want to get my email too, so let’s not make our servers crash by keeping the email that needs to be kept on the server itself.  Let’s move it. BUT … not ALL of it, for goodness sakes!!  And be thoughtful about where you move it to, and why.  An email archive is nothing more than a dumb repository with a search engine.  It won’t look at your email and make a determination of whether or not it should be kept; it just keeps it.

Email needs to be evaluated based on its content, then deleted when it’s not a record and kept when it is.  If you do this, consistently and pervasively, I guarantee the weight on your servers will be far less than before and may find you don’t need an email archive in the first place. 

However, if you are a large organization with extraordinarily large volumes of email and just can’t let it sit on the server for, say 30, 60 or 90 days waiting for an end user to make a determination of the record status of their email, then fine, go get an email archive system, but know this …

Email archives are/should be temporary storage for mail that simply can’t sit on the server due to operational efficiency.  In the end, email should either be deleted or moved to a system that will allow the organization to manage email records as records throughout their retention period.

Email archive systems are not records management, document management or content management systems. 

When you send email there, understand that it should not live there forever as a way to meet your compliance obligations.  If you need to retain email as a corporate record, it is far better off, in the long term, being stored in a system where email is organized according to your retention schedule/corporate taxonomy and which can apply retention rules to the content. (Note: Some email archiving vendors claim their products can do this; however I have yet to find one that can natively manage email as a record as well as a records management system can – based on records management criteria such as records series, etc.  If you are such a vendor and want to prove me wrong, please comment on this post – we want to know who you are!).  That’s not to say that email in an email archiving system CAN’T be managed according to a retention schedule, it’s just hard(er).

The compliance piece comes into play because when you employ an email archiving system, you are essentially centralizing the storage of email (usurping the creation of .psts and other off-network storage).  When email is in a central repository, you can search it for compliance (and ediscovery) needs, and generally produce relevant email.  However, again, if you wholesale off-load your mail server into an archive, you are simply increasing the amount of stuff you have to search through for compliance/discovery efforts.  Don’t move it all – it’s not worth it!

Did I make my point well enough?  DON’T MOVE ALL EMAIL TO THE EMAIL ARCHIVE – MANAGE YOUR EMAIL *THEN* ARCHIVE IT!

Oktxhbai.

Side note: Is it email or emails?  I asked that question on Twitter and there appears to be a great divide, and some folks are quite passionate about their choice.  I go back and forth, but generally use email as both singular and plural, and as a verb.  Comes down to one less keystroke per instance, and I like that economy.

Report

Rate Post

You need to log in to rate blog posts. Click here to login.

Add a Comment

You need to log in to post messages. Click here to login.

Comments

Douglas Schultz

Julie - you are right on with your comments. It depends on who is running the project as to whether the email managed like all other content or the archive becomes just another dumping ground that someone has to go through later. Too many times it is another digital landfill.
Report
Was this helpful? Yes No
Reply
Julie Colgan

Thanks for your comment Doug. It's nice to know I'm not alone in my frustration with the way these tools have generally been deployed ... creating "just another dumping ground". Hopefully I will be able to help my client make the most of their investments by using the tool they purchased in a smart, proactive way.
Report
Was this helpful? Yes No
Reply

Kaprice D

Julie,

You bring up some interesting points that mirror the frustrations we hear each and every day from our clients. Moving emails from one solution to another does not make them compliant, help to follow retention schedules, or make emails more accessible.

Your statement "Email archive systems are not records management, document management or content management systems." is right on the money. Many of our clients are in the financial industry and we have been working with them for years developing solutions to help with their compliance needs.
You were looking to hear from a vendor’s perspective on how email archive can be achieved in a balanced and compliant format. I would like to share with you what our company has developed based on industry information and feedback and concerns from our clients.

We have developed an email archive solution that allows for emails to be searched based on key words as well as by domains, users, date ranges and with the use of full text. You also mention many email archive solutions are just another location for emails to be stored and that is not enough to help meet retention schedules. To collectively combine the needs of email retention as well as document retention we have developed a solution integrating our email archive platform with a document management solution which treats an email just as it would any other document in a document management repository allowing the necessary retention schedules, security and indexing necessary to meets compliance needs.

With today’s ever changing needs and with the use of so many electronic platforms it sometimes can become very overwhelming to most when they now have to become responsible for tracking not only their paper but also their electronic forms of communication. Even for those of us who may be more tech savvy it comes down to which tool is really right for the needs of the company.
Report
Was this helpful? Yes No
Reply
Julie Colgan

Hi Kaprice - thanks for taking the time to respond. I'm happy to hear that vendors are actively listening to their customers and taking note that the tools, as traditionally developed and deployed, don't really cure the disease, they just put a bandaid on it.

You mention using a DMS for the storage of email along with other content that needs to be kept, and that is an approach that I would consider a best practice (so long as it is a repository that has RM functionality), however ... I do think that email archives could be more useful if they would figure out how to pick up metadata from email systems based on user organization.

For example, if I set up a sub-folder in my Outlook Inbox called "Project Horseshoes", I want that to carry over to the email archive and for any email associated with that container to be deposited in the same spot as they move to the archive over time. That way, later, if I have to manage the retention and disposition of email in the archive itself rather than it being moved to some other repository, I can at least have the context of a group of documents in tact. Context can be far more valuable than most generic email metadata like To, From, etc.

Just my thoughts on the issue.

Thanks again for your contribution to this thread!

Julie
Report
Was this helpful? Yes No
Reply

Martin Tuip

Finally got my password so that I could comment. Trying not to sound to harsh as there areas where you have a point I'd like to say a few words. I've been in the archiving industry for almost a decade and have seen it grow from 3 vendors to over 100 email archiving vendors. Yes .. there are more than 100 vendors that offer email archiving in one way or another and with that comes the first problem.

1. Not all solutions are alike.

Archiving products take enormous development resources to build and because of that .. some products will have far less functionality and could be of a huge frustration to the users who's solution is forced upon them. Some products in the market actually don't offer retention at all, whilst others can go into extremes on this. Without knowing the products the users are frustrated with .. it could very well be one that is on the lower end of the scale.

2. IT will deploy archiving without working with RM

I've given tons of webinars and seminars about this topic and I have mentioned it in many. IT will deploy an archiving solution or something similar without RM .. and its part of RMs job to ensure that I understands what they do and their responsibilities. The world is changing and you have to be proactive here and work together with them to define the requirements. If you don't .. well .. your organization might not buy what you think is needed.

3. Email might very well be too high of a volume to manually classify

With probably the top 5 solutions out there it is extremely easy to setup classifcation in such a way that end users are capable to classify content and based upon classification only keep relevant information. Things have drastically changed since 2005 and there are multiple ways to do this (whether it is through a folder structure or even through keywords). Maybe IT didn't set this up properly. Now .. I'm of the opinion that in many cases email might be too high of a volume to classify every single message. I send and receive several hundred messages per day .. and if I add it all up .. I probably receive more than 1,000 email messages on my accounts combined. No .. I don't read them all, but if I would have to classify them .. I would spend my day reading my email instead of being productive. So the majority of organizations that I have seen implementing email archiving solutions have been deploying group based policies where in combination with limited manual classification, the rest of the data gets handled together. You work in finance thus most likely all your data is finance related give or take a few messages here and there.

4. An email archive isn't just a dumb box witn an index

First of all .. within archiving vendors .. not all indexes are alike. Some can only index a few dozen content types, whilst others can index hundreds of content types. And if you can't search on it .. why keep it. Second is that quite a few email archives are part of a wider solution that can handle tons of data sources like SharePoint and File data.



Lets keep talking :)
Report
Was this helpful? Yes No
Reply
Julie Colgan

As rants often do, they incite some good conversation. Thanks for the comment, Martin!

I concede, perhaps I was a bit over-generalized in my orignal post and am happy to give credit where credit is duly due. Mr. Tuip and I had a nice conversation offline and I learned that the product he works with, NearPoint, does in fact do what I mentioned in an earlier comment - pick up user-based classification from the inbox and move it to the archive. I appreciate him clearing that up.

The invitation from my original post still stands: if there are other products that do that, I invite you to share with us.

Ok, so as a reply goes ... I'll now take a stab at replying to the points you made:

1. True. Due diligence is on the shoulders of the purchaser. The frustration I have experienced myself and have heard about from others is typically either due to a poor product or being brought to the conversation after a poor implementation of even a good product ... which brings us to #2 (where the majority of the existing problems with email archiving lie, IMO) ...

2. True, again. IT does deploy without RM involvement. Often. I hear you that RMs need to shout from the rooftops that they are there and should be included in the strategy, puchase and implementation of these systems, however that has seen varied success. IT has the budget and often the authority to move forward without seeking comment from others. In some instances, even if the RM makes their presence and expertise known, that doesn't automatically mean they are invited to join the effort. If the point of the purchase and implementation is to alleviate system weight, the RM may be seen as an obstacle (or at least a delay) to getting the job done. Dealing with content is hard - IT knows that.

So, for as easy as it is to say email archiving gone awry is the RMs fault for not insisting they be allowed to participate, it is perhaps just as much the fault of IT for not proactively seeking out experts in their organization or taking the time to worry about content as much as they worry about system efficiency. I'd also add that perhaps the vendor and/or integrator should be in the mix of responsible parties here. Sound advice about the consequences of short-sighted implementation strategy based on previous experience and best practices is something the vendor can bring to the table as well. (I'm not implying that any particular vendor does/doesn't provide such advice - it's merely a statement that all parties have a responsbility to work together to ensure the best outcome.)

3. As yes, classification. My favorite! I agree that manual classification of every email may not be possible, but that isn't the point I was making. My point is that archive systems should leverage whatever efforts are already being expended on classification efforts by the end user. As I mentioned above, I have learned that at least one product offers this functionality.

The value here is, even if users aren't reading every email (which I'm sure they aren't - who does?), they *are* reading many/most of them, and acting on them. Every email user with any volume of email hitting their inbox makes some effort to organize that content. If their company doesn't push classification to them, then they make it up for themselves. That is valuable structure and context that absolutely should continue to be leveraged when/if mail is moved to another repository.

The point about group-based policies may work for *some* parts of an organization, however it won't work for all of them. For example, as a consultant, I work on a variety of projects for clients that don't all have the same retention period (mostly because they are all trigger based ... but I won't go into that here). I'll just leave it at ... if you're going to use user-role as the basis of your email management strategy, understand that it may not work well for some portions of your organization, particularly those business units who work mainly on project-based tasks like Legal, Construction, etc. Retention based on creation date just won't do.

4. Think I hit a nerve there ... and perhaps rightfully so, for *some* products. However, even the best product implemented poorly is still little more than a repository with a search engine (or a dumb box with an index - whichever you prefer).

In the end, I don't think I've heard anything that negates the main point of the original post - that organizations should manage their email before they archive it. Or maybe it's better stated, an organization should have a strategy to manage their email before they buy and implement an email archive system ... :)

Thanks again for your reply, Martin. I appreciate your taking time to share your insights - I know I learned something new!

And I agree, let's keep talking ...

Report
Was this helpful? Yes No
Reply

David Bruce

Hi Julie,

I have been thinking about this recently. Seems to me there are three different concerns:

1. A compliance archive is required.
Solution: Journaling. The Email part of this could be achieved by storing all mail in and out of the company. Access wold be restricted to a few legal / HR staff, who could in principle see all records. A full-text indexed Discovery / Search function would be needed. Such an archive would rarely be used, but would give a reasonable level of confidence that relevant emails for a compliance issue could be found.

Retention periods should be enforced. Simplest would be to set a fixed retention period for all types of mail in this archive - for example 10+1 years if the main requirement for retention of financials is 10 years. Does this cost storage? You bet - but the business case assessment should be simple enough to decide at a top management level.

2. Operational "Inbox management" is required.
Solution: Automated off-loading of mailboxes to reduce mail system storage.". Tools are available to migrate mails out of the main storage in the mail system, yet keep transparent access to the mails from your mail client (e.g. Outlook) - including search across the different storage types.

Retention periods should be defined. Could be e.g. 1 year. Any retention after that must be based on explicit filing. Busines case is partly technical, partly convenience for users.

3. Business documents have to be filed.
Solution: Content Mangement. Explicit filing of documents with relevant metadata, with IT-supported tools to minimise the manual effort.

Retention periods are defined based on the retention needs for the different kinds of content - can also be the context that defines the retention, rather than just the information per se. Business case is as always difficult when talking ECM, but necessary!!

In all cases, data to be deleted when retention periods have expired.

By separating these three different requirements, there are three different types of solution, each with their business case for decision makers.

Of course you could look for hybrid solutions that could perhaps handle all of these scenarios. If they work with separate repositories for the different scenarios, then this is just a question of product packaging. If there is a robust solution that can do it all with one mail repository and no increase of manual or admin work, I would like to see it! Currently I think that if you want to cover all 3 scenarios, then you should look for 3 solutions. More product cost, but less operational and user-process complexity. Perhaps!

In our company (Europe-based, international scope, approx 12,000 employees), we currently use Exchange and Documentum. We expect gradually to migrate content management to a managed set-up in Sharepoint. We will investigate journalling in Exchange 2010 to see whether this will be sufficient for the compliance archiving. We have seen one tool for inbox management, and will consider the business case for such a tool.

I look forward to hear your comments / feedback / great ideas!

Best regards,
David Bruce (Enterprise Architect)
Report
Was this helpful? Yes No
Reply

Julie Colgan

Hi David - thanks for your contribution!

Personally, I am wary of enterprise journaling. Depends on the organization though. Journaling may have a nice place in a disaster recovery plan, be useful for a handful of particular users, or appropriate for the mostly highly-regulated and/or highly litigious organizations; but across the board journaling generally opens up all kinds of risk. In any use case, it is my opinion that journals should have a fairly short retention period.

Compliance means that someone says you need to keep some content for some period of time (not all content, just some content, and generally not forever either).

Discovery means you need to find relevant stuff you have/should have and produce it.

If you employ records management principles to managing your content, compliance can be achieved. You know the things you need to keep and how long you need to keep them, those things get kept and get kept in a way they can be retrieved as needed, and then are disposed of when their value has been met.

And if you do compliance well, then discovery becomes much easier and much more defensible. You know what you have, where it is (or when and why it was disposed of), can suspend disposition as needed for litigation, can produce the relevant bits, and then dispose of them when appropriate.

If you want to employ journaling as a stop-gap measure while you develop a full-fledged strategy, that might be okay; but again, consider the fact that you will likely be over-retaining and may be out of compliance with your own retention policies. Just because the journal content is only accessible to a certain few in your organization does not protect it from discovery in litigation.

Your #2 is the basis of my original post and is where email archiving is generally deployed. The point here is that organizations need to manage their content, not just throw technology at it. There are lots and lots and lots of ways to go about balancing server efficiency, users' need for flexibility to work the way they need to work, compliance requirements, risk management, etc. The type and amount of automation involved in "Inbox Management" will be dependent upon all of those factors.

An example from my experience ...

An organziation had a content management system (CMS) and were using Exchange. The CMS was purchased and implemented without RM involvement, but honestly they had done a pretty good job with it.

Users were encouraged to file email to the CMS, but not required to do so (this was the real failure of the CMS implementation - policy, not technology). The result was that there was still a lot of weight on Exchange because some users didn't want to move stuff to the CMS (even though they could still "see" it within their Inbox). That is when the email archive system was purchased.

When the email archive was implemented, it was implemented with a 120-day auto-archive period and an "infinite" retention period (I wasn't there when those decisions were made so can't say why, but imagine it was because getting to the real answer of how long to retain email content that had lost its context was a hard answer to come by).

Once I got into the mix, we (RM, IT and management) agreed that our strategy should be for users to file email that were records to the CMS. We would continue to allow for 120-day archiving to alleviate weight on the server and keep everything forever, until such a point as our organization got better at disseminating and following and enforcing the policy of filing to the CMS.

Once we were better at filing to the CMS, we would be comfortable setting one retention period for the entire archive content (that was a heated debate - I wanted 1 year and IT wanted 5 years. I conceded but felt it was still a victory over "infinite"!). Basically, anything that hit the archive would have the generic retention period applied which would give the user time to file the email to the CMS (move it from the archive to the CMS). Anything that hadn't been filed by the end of the retention period got blown away.

This strategy still fixed the immediate problem of too much weight on the server, but also moved us to where we needed to be in terms of policy and content management. It also allowed us to plan to re-use the storage platform that was intended for mass email storage for other purposes with the expectation that the volume of email that ended up hitting the archive would be less and less the better we got at managing email proactively.

So to your point #3, yes, the ultimate end result may be a mix of technologies working together to achieve the desired outcome. The keys to success are 1) knowing what your desired outcome actually is, 2) understanding your risk tolerance, compliance and litiagtion environment, user expectations/needs, your culture (particularly in terms of your success at implementing, following and enforcing policy) and your technology environment and 3) doing good due diligence when chosing products to ensure you meet #1 (your desired outcome) within the constraints of everything in #2 (risk, compliance, needs, etc.).

Best of luck to you in your quest to find the right answer for your organization. I hope at least some of this will prove helpful to you! And check back in to let us know how it's going!



Report
Was this helpful? Yes No
Reply
Amin EL-Assaad

I hope readers continue reading your comments ...

I wish to thank you for raising this subject up and especially in your rant fashion :)

The picture got much clearer at this point of the discussion, thank to your honest replies ... I really enjoyed reading this thread ...
Report
Was this helpful? Yes No
Reply

Martin Tuip

Journaling only captures partial data and doesn't capture everything that you probably would like to see. For instance it only captures meeting requests and not manual calendar items, tasks, contacts or other data that might be written in the mailbox including end user classification.

imho Journaling isn't the answer, more of less one of the tools you have in a tool box to do things.

All 3 points can be done with 1 solution though however it might require you to leverage SharePoint and Exchange as front ends with the appropriate configuration and archiving solution. Again .. you have to do your due diligence and research the vendor solutions to see what can be done. Like I said before .. there are 100 vendors offering solutions and not all are of the same caliber.

Instead of stubbing or seamless access to email I've been long a proponent of age based data access. Do you keep your tax records on your kitchen table for 7 years or do you file them away and you know where to find them. The same counts for email .. do you really need to keep all email, even those from 5 years ago, in your inbox .. of you facilitate a way to find them (knowing that 99.9% of email older than 6 months is never looked at again).

Report
Was this helpful? Yes No
Reply

This post and comment(s) reflect the personal perspectives of community members, and not necessarily those of their employers or of AIIM International