[Netidentity] Email Migration Update... Update

As promised, I’ve got an update for you coming out of our afternoon meeting.

1. Mailbox index corruption. I have been informed that this problem has been solved. We were able to ascertain that the indexes of roughly 2500 accounts had been corrupted. Once all of the accounts were identified, it was relatively easy to fix the issue. This means that no one should be stuck on the pending migration page.

2. Domains not provisioned. We are making progress on solving this issue. We have narrowed the range of affected users significantly and expect to have this issue completely solved early tomorrow.

3. System Slowness. The team doubled the capacity of the system (technically, the “PS Nodes”) from 8 to 16. Originally, they were only going to go to 14, but were able to swing some additional resources at the last minute and go right to 16. It appears that this capacity increase has solved the performance issues that we were experiencing earlier today.

There are still a number of other outstanding issues which affect a much smaller number of subscribers. The team is aware of these and working on them as well – here is a run-down of those other issues as well;

Subscribers have reported that they cannot login to their webmail. This was related to the system slowness issue. If you are still experiencing this issue, please ensure that you notify our support team.

Subscribers get “time-out” messages. This was also related to the system slowness issue and should be completely resolved as of this writing.

Subscribers continue to receive “migration pending” notices for a period that exceeds the length of their actual migration. This was related to the mailbox index corruption issue and should be completely solved now. If you are still experiencing this issue, please let us know.

POP service isn’t working. This was also due to the corruption issue and should be completely fixed now.

Duplication of messages in various folders. Also related to the corruption issue and fixed.

Customers not receiving email even though they have verified that it has been sent. Also related to the corruption issue and also fixed.

Folders are disappearing after the migration is over. We are in the process of running a repair job that will make your folders reappear. Most users folders have been dealt with, but there are still some outstanding. Resolution of this specific issue is still pending.

Other users folders are visible. We are still investigating this issue. So far, we have been able to determine that some subscribers mailboxes include email folders that they did not create. So far, none of these folders have included any actual mail. While we haven’t completed our investigation, we have not been able to determine that actual user mail was exposed to any third party during this transition. I will post an update to this issue as soon as we know more.

Deleted messages are re-appearing. This has been reported by a very small number of subscribers. We will address this prior to completing the migration.

We have another status meeting at 10:30 EST tomorrow morning at which point I’ll know exactly how we’ve fared in dealing with these various issues overnight. In the meantime, we will continue to work on the problem, answer tickets, email and so on.

I’ve been reading through the various comments and criticisms on the blog. I can only say that all of you are a remarkably patient and helpful group. I really appreciate all of the feedback and comments – it has been very useful in helping us diagnose and deal with various issues. Don’t be surprised if you receive an email from me offline looking for more information about the trouble you’ve described here. Some of the more esoteric issues require some special handling and I’ve really appreciated those of you that have taken the time to give us the information we need to solve the problems you’ve been describing.

Thanks again.

Comments

[T]

I have a great idea!

Why don't you give us back the old webmail system, and TEST INTERNALLY (or TEST AT ALL) the new one! In about 5 years time everything should be working :)

Bugs/errors/etc.:
- "Compose" button doesn't work. Hillarious! I can't write a new mail in my webmail! That's mindblowing! Going to be joke #1 this year in my office.
- Still NO OPERA BROWSER SUPPORT. I can't take IE much longer...
- I'm getting "your host is not welcome" from more and more mail-servers as a response to sent mail to them :(
- GUI is getting uglier more I use it :/

I found 1 positive feature - system is working a bit faster than in the morning. BUT it's still FAR, FAR AWAY from what it used to be.
I guess Tomcat wasn't the best idea.

Pozdrawiam,
Tomasz

[chris]

Well, let's see. The support page does not work, so I cannot send a support letter. All my filters are gone. The convenient button to select the next page of mail is gone, replaced by the clumsy pull down menu.

Reporting spam has become a process requiring multiple steps in place of the button to click, report, and delete the spam.

When I try to forward mail to abuse@netidentity.com, I get frequent errors that the mail was not sent. The GUI is worse than the old one; and is more awkward to use. The choice of colors must have come from New York City, it is so ugly.

Some allegedly improved features have been included in the package at a higher price. The e-mail package is advertised at a price lower than the cost of renewing the existing service which I have had since the earliest days of Mailbank.com. Unfortunately, I had already renewed at the higher price.

On a positive note, the mail process still works. I guess that I can thankful for small favors.

I am hoping that the new system will eventually become as useful and responsive as the old one was.

Chris

[Carsten]

Um, you say that the mailbox index corruption has been resolved, but my POP client (Eudora) says otherwise, and my webmail is still stuck on the migration message... Resolved? Not exactly!

Furthermore, when I call the toll free support line I either get no answer at all or a busy signal. At least this morning I was put on hold for half an hour before I had to give up... Customer support?? Not exactly!!

I have now been without my email for two days. This is completely unacceptable! Did you guys do ANY testing before attempting this migration? Don't you have mirrored servers that you could switch back to when it became apparent that the transition was not going well? And would it have been too much to ask to send out a proper set of pre-transition notices to your users warning them of the transition and the risk of service interruption? This is all standard operating procedure for a well-run iT shop... But NetIdentity??? Not exactly!!!

Needless to say, I'm not a happy camper...

[Writingmom]

I am appreciative of the update. Although I can get to my mail on the web, I still do not have POP access. The message above indicates it should be working.

Does anyone else still have this problem? Did anyone figure out a fix to the problem? I'm trying to determine if there is anything I can do from this end but I haven't seen any information on that.

[Ross]

Carsten - I just got off the phone with another user experiencing the same issue as you. I've escalated to the right people internally here and expect a fairly quick analysis, but I can't be sure what they are going to say as far as fixing the problem goes.

This is definitely *not good* - be assured that we will deal with this for you.

Regarding our pre-transition notices, we did send out communications, but we were overly confident that there would be no service interuption. I wish we had assessed this differently.

-ross