Category Archives: Exchange Server

Exchange Server

Exchange Move Mailbox logon info

Leave a comment

Event IDs related to Mailbox moves

The Event source for Mailbox moves is Exchange Migration, and the events are logged in the Application log on the server where an administrator started the move. Meaning; where the admin was logged on and used the EMC or EMS to start the mailbox move. It would be nice to have all the mailbox move events gathered in one place, and that can be performed with Event collection.

Event ID Level  
1006 Information Indicates the start of a mailbox move
1022 Error Logon failure on database, caused by the mailbox move
9660 Warning User X failed to log on because their mailbox is in the process of being moved
9873 Information A named property has been created for database XX. Meaning the new mailbox has been created in the destination database.
1007 Information The mailbox was successfully moved.
9354 Information The mailbox is deleted from the source database
1154 Information Rule synchronization between old and new mailbox has completed successfully.
1205 Error Failed to create mailbox

Last logged on user

Whenever you move a mailbox in Exchange Server the LastLoggedOnUserAccount attribute of the mailbox is changed to the user running the mailbox move. This is confusing since that user does not need permission to log on to the mailbox to perform the move.

It can also present a problem if you are trying to figure out which users have yet to log on to their mailboxes after moving them between servers. Let’s say you run this command:

Get-MailboxStatistics | where { $_.LastLogonTime -eq $null }

This would show you which users have never logged on to their mailboxes. Whichever servers they originally were located on cannot be removed before they have logged on at least once and been redirected to their new mailbox servers. Unfortunately, since the user performing the mailbox move is added to the LastLoggedOnUserAccount this command will probably return no users.

Exchange Server

Configuring language settings for Out of Office messages in Exchange 2007

Leave a comment

Exchange provides the ability to configure Out of Office messages. The owner of a mailbox can configure different messages to be sent to users inside the organization and outside. The user is free to enter any text he wants, in any language, but the subject of the Out of Office message i handled by the server. Typically the Out of Office message you receive when sending mail to a mailbox that has Out of Office messages enabled is Out of Office: <subject of message sent to the mailbox>.

The Out of Office phrase is used by the server because the languages attribute of the mailbox that has Out of Office messages enabled is English (en-US). But you may also see something like this, if the language of the mailbox has been changed:

I this case the phrase is Fraværende, which is the Norwegian translation for Out of Office (or close). The mailbox using Out of Office in this case has its language set to Norwegian (nb-NO). I you want to change the language settings for a mailbox, there are two ways (that I know of):

  1. The user can change the language from within Outlook Web Access:
  2. The administrator can change the language from the Exchange Management Shell (EMS):

    Command for a single mailbox:
    Set-Mailbox morgan –Languages nb-NO
    Command for several mailboxes:
    Get-Mailbox | Set-Mailbox –Languages en-US
    A full list of all language codes is available here, but I do not know if Exchange 2007 support them all.

A little note about the Languages attribute. It is an array that can actually hold several values. I have seen mailboxes with two languages set here, but I am unsure of how this has come about. I am guessing that something has been set in OWA, but that some setting(s) still remain in a different language. I am not sure how this affects Out of Office messages.

Also to note is that in my lab I had to restart both the Microsoft Exchange Active Directory Topology Service and the Microsoft Exchange Information Store services to make the change take affect. This is probably caused by the Exchange Directory cache.

On a related note, look here for ways to change the language of DSN messages. There are two settings, one for internal DSNs and one for external.

 

 

 

For another post about the workings of Out of Office look here.

Exchange Server

Exchange Out of Office replies

Leave a comment

I guess you are all familiar with the Out of Office Assistant in Exchange server. Whenever you activate it Exchange will send a messages to anyone who sends you mail that you are out of the office. I never really thought much about the internals of OOF Assistant, but a question from a customer prompted me to do some digging. The question was simply about how often the OOF Assistant would send messages, or Out of Office replies, as they are called. I had no idea and had just assumed that OOF Assistant would send a reply for every message received, but a quick test revealed that that was not the case. It turns out that OOF Assistant maintains a table of all senders sending you mail when the OOF is activated. It will only send an OOF message once to each sender. So if I am away for two weeks and you send me 100 messages (from the same account) you will only receive one OOF messages from me. Neat! More info about this is in this KB article:

Only one reply is sent to each sender when the Out of Office Assistant is enabled (http://support.microsoft.com/kb/157961)

Exchange Server

Outlook Web Access: There was an unknown error accessing the directory

Leave a comment

While troubleshooting message delivery between two email systems, I received this error from Outlook Web Access:

—————————
Message from webpage
—————————
There was an unknown error accessing the directory.  If the problem continues, contact technical support for your organization.

I was trying to send a message to contact in my Exchange organization representing a user on the other messaging system. This particular contact was hidden from the address list so I was manually typing the address. I was sending to the targetAddress, or External E-Mail Address, of the contact, not the local address.

A quick web search did not reveal an answer. The only KB from Microsoft regarding the error, http://support.microsoft.com/kb/945917, was not the solution in my case.

First I had a look at the email addresses of the contact. It had only one; userx@sub.domain.com. (domain.com is a shared domain between the two messaging systems.) The address was both in the proxyAddresses and targetAddress attributes. Since sub.domain.com is a sub-domain of the shared SMTP domain I figured I would try adding an address from domain.com. That is when the solution presented itself, because Exchange Management Console (EMC) displayed this error:

——————————————————–
Microsoft Exchange Error
——————————————————–
The following error(s) occurred while saving changes:

set-mailcontact
Failed
Error:
The proxy address “SMTP:userx@sub.domain.com” is already being used by “domain.com/Users/userx”. Please choose another proxy address.

Turns out that there was already a mail enabled user in the directory using the address. That meant that when OWA tried to resolve the address the directory returned two matches. That is one too many. After I resolved this conflict the error disappeared from OWA.

Exchange Server

Licensing your Exchange 2007 Edge Server

3 Comments

One of the new roles in Exchange 2007 is the Edge server role. The edge server is an Exchange server that sits in your DMZ and receives and sends SMTP mail to and from the Internet. It also performs a lot of other task like spam checking, anti-virus (if installed) etc. The Edge server is not a member of you Active Directory domain and uses an ADAM partition to hold the directory information it receives from an internal Hub/Transport server. The Edge server is present in Active Directory though, it is listed with the other servers under the Exchange Administrative Group (FYDIBOHF23SPDLT) in the configuration partition.

Whenever you install any Exchange 2007 server, before you enter a product key, each time you open the Exchange Management Console (EMC) you are reminded of the Exchange servers that still do not have a product key registered and are thus regarded by the EMC as trial servers. You can enter the product keys by using EMC or the Exchange Management Shell (EMS) which is PowerShell with some additions. I recently ran across an interesting problem trying to enter the product key for my Edge server.

After entering the product key for my combined Mailbox, Hub/Transport server and the CAS server; they disappeared from the warning displayed when I opened the EMC. But that still left me with the Edge server. First I tried entering the product key on the Edge server itself, that succeeded and the warning disappeared from the local EMC on the Edge server, but since there is no communication from the Edge server to the Hub/Transport server, only the other way, that did not help with the error displayed in the EMC on the internal servers. Next I tried using the EMS to set the key on the Edge server. The command is:

Set-ExchangeServer <servername> -ProductKey AAAAA-BBBBB-CCCCC-DDDDD-EEEEE

I then received this error:

Set-ExchangeServer : An error happened while accessing registry of the specified server: “<servername>”. The error message: “Attempted to perform an unauthorized operation.”.

At line:1 char:19

+ Set-ExchangeServer <<<< <servername> -ProductKey AAAAA-BBBBB-CCCCC-DDDDD-EEEEE

 

I guessed that the reason was that my domain administrator account was not allowed to update the registry on the Edge server, which made sense. (I did open port 445/TCP from the H/T server to the Edge server in the firewall, in addition to the ports required for ADAM sync.) I then tried running the cmdlet using runas in different ways, using the NETONLY option etc. If I ran runas with NETONLY I could access the Edge server, but then I lost access to the DC since the local account on the Edge server, which runas was using, did not have access to Active Directory. After pondering this duality for a while, the need to access the Active Directory and the registry on the Edge server at the same time over the network, using only one account I figured it out. How do you get “single-sign-on” in Windows across different security boundaries? You create duplicate accounts in each security boundary and give them the same password.

I created an account in the Edge Server’s security database (SAM) with the same name as my internal Domain Admins account and gave it the same password. Immediately the Set-ExchangeServer cmdlet succeeded and my Edge server was licensed in the eyes of Active Directory. No more warnings in the EMC.

I have installed a few Exchange 2007 servers now, but never had this problem before. That was also the reason it took me some time to figure this out; it had never been a problem before! I kept asking myself “Why doesn’t it work at this site, when it has worked so many times before?”. The reason was that on all the other sites I was already running with identical accounts on the internal network and on the Edge server. But this time the accounts were different.

Exchange Server

Mismatched Email addresses causes recipients to be skipped during Offline Address Book generation

Leave a comment

You might experience this error during an Offline Address Book generation cycle:

Index : 94460
EntryType : Warning
EventID : 9327
Message : OALGen skipped some entries in the offline address list ‘Global Address List’. To see which entries are affected, event logging for the OAL Generator must be set to at least medium.
– Default Offline Address List
Category : OAL Generator
CategoryNumber : 13
ReplacementStrings : {Global Address List, Default Offline Address List}
Source : MSExchangeSA
TimeGenerated : 03.01.2008 11:35:31
TimeWritten : 03.01.2008 11:35:31
UserName :

After you set the event log level for the OAL Generator to at least medium, eg. by using this EMS command:

Set-EventLogLevel ‘MSExchangeSAOAL Generator’ -level Medium

You start seeing these errors:

Index : 94454
EntryType : Error
EventID : 9325
Message : OALGen will skip user entry ‘user1’ in address list ‘Global Address List’ because the SMTP address ” is invalid.
– Default Offline Address List
Category : OAL Generator
CategoryNumber : 13
ReplacementStrings : {user1, Global Address List, , Default Offline Address List}
Source : MSExchangeSA
TimeGenerated : 03.01.2008 11:35:27
TimeWritten : 03.01.2008 11:35:27
UserName :

As we can see from the error in the Event Log, OAL Generator claims that the SMTP address ” (blank) is invalid. This is not surprising, as a blank address can not be used for anything.

I have discovered one reason for this error, there might be more. If the user’s primary SMTP address does not match the value in the mail attribute in Active Directory, this error is generated. This happens if you change the primary SMTP address in EMC. EMC does not update the address in the mail attribute. To see if you have any recipients in your organization that have a mismatch between these two values, run these EMS commands:

get-mailbox -resultsize unlimited | where { $_.WindowsEmailAddress -ne $_.PrimarySmtpAddress } | ft –auto

get-distributiongroup -resultsize unlimited | where { $_.WindowsEmailAddress -ne $_.PrimarySmtpAddress } | ft –auto

get-dynamicdistributiongroup -resultsize unlimited | where { $_.WindowsEmailAddress -ne $_.PrimarySmtpAddress } | ft –auto

This should be possible to do with Get-Recipient as well, but I cannot make it work. Get-Recipient always return every recipient in the organization.

To remedy this situation, these EMS commands may be of interest:

get-mailbox -resultsize unlimited | where { $_.WindowsEmailAddress -ne $_.PrimarySmtpAddress } | ForEach { Set-Mailbox $_ -WindowsEmailAddress $_.PrimarySMTPAddress }

get-distributiongroup -resultsize unlimited | where { $_.WindowsEmailAddress -ne $_.PrimarySmtpAddress } | ForEach { Set-distributiongroup $_ -WindowsEmailAddress $_.PrimarySMTPAddress}

get-dynamicdistributiongroup -resultsize unlimited | where { $_.WindowsEmailAddress -ne $_.PrimarySmtpAddress } | ForEach { Set-dynamicdistributiongroup $_ -WindowsEmailAddress $_.PrimarySMTPAddress}

You should probably test these commands with the –whatif parameter added to the Set cmdlets.

A good pointer as to which recipients have a mismatch between these values, are those recipients who no longer have their Email addresses updated by a recipient policy. You can quickly list which recipients are in this state:

get-mailbox -ResultSize unlimited | where { $_.EmailAddressPolicyEnabled -eq $false } | ft –auto

get-distributiongroup -ResultSize unlimited | where { $_.EmailAddressPolicyEnabled -eq $false } | ft –auto

get-dynamicdistributiongroup -ResultSize unlimited | where { $_.EmailAddressPolicyEnabled -eq $false } | ft –auto

Also recipients who are targets of E-Mail address policies (EAP), but where those policies have not been applied, are candidates for this error.

Lastly, you cannot set the mail attribute if a recipient is a target of an EAP.

Remember to set the Event Log level back to it’s original value after you have finished troubleshooting:

Set-EventLogLevel ‘MSExchangeSAOAL Generator’ -level lowest

Exchange Server

Using IIS 7 URL Rewrite Module to simplify Exchange 2007/2010 Outlook Web Access URLs on Windows Server 2008/2008 R2

11 Comments

The default URL for Outlook Web Access i Exchange Server 2007 is https://<server FQDN>/owa. This URL is always an issue for end users who find it too long or complex. They either forget to use HTTPS in front of the URL, or to append /owa to the server name, or both. The first mistake results in an error saying SSL is required. The second loads the root of the site. I agree that this could be made more user-friendly so I always implement some form of rewrite or redirection. Typically I make OWA available at just <server FQDN>. No HTTPS or subfolder, e.g. owa.company.com. This requires two things; redirecting HTTP traffic to HTTPS and redirecting the root folder to the /owa subfolder.

Note: The final URL for OWA is always (almost) https://<server FQDN>/owa. We cannot disable HTTPS or publish OWA at the root of the site. But the URL users type can be simplified. This simplification is what I mean when I say redirect in this article.

Until recently the redirection to HTTPS and the subfolder OWA could be accomplished by first redirecting the root folder with IIS 7’s HTTP redirect functionality and then editing the HTTP error page , typically 403, so that it would redirect to HTTPS instead of throwing an error. I have never liked this approach, especially the latter part. You could also use ISA Server 2006 in front of your Exchange server and do the same there. But recently a much better solution has arrived. The IIS team has released the URL Rewrite Module extension to IIS 7 which lets you perform advanced URL rewrites and redirects using, among other things, regular expressions.

How to simplify the OWA URL with URL Rewrite:

      1. Download and install the URL Rewrite extension on your CAS server.
        http://www.iis.net/extensions/URLRewrite
        Remember to get the version for your architecture. This will almost always be x64 unless you are in a lab and running the x86 version of Exchange 2007.
        To avoid having to restart your server follow the steps in this post:
        http://forums.iis.net/t/1153276.aspx
        Also, these instructions are written for the 2.0 version. I cannot guarantee that they will work with any older or newer version.
      2. Disable the Require SSL setting on the Default Web Site.
        This is necessary for the redirection in URL Rewrite to work. This is not a security issue since URL Rewrite will force SSL for the entire site (except for OAB).
        040209_1209_UsingIIS7UR1
      3. Open the web.config file under your wwwroot folder.
        This is usually under %systemdrive%inetpubwwwroot.
        The web.config file does not exist by default, so change a setting on your site and change it back again to have IIS generate the file.
      4. Paste the following text in web.config in the <system.webserver> section:

https://gist.github.com/morgansimonsen/8040092

NOTE: The PowerShell vdir entry is to exempt the /powershell virtual directory found on Exchange 2010 Servers from HTTPS enforcement. This is necessary for remote management of Exchange 2010 Client Access Servers. This rule should have no bearing on an Exchange 2007 server as they do not have the powershell vdir.

This will create three new rewrite rules.

      • Open the URL Rewrite item under Default Web Site.
        040209_1209_UsingIIS7UR2
      • Your three new rules will look like this:
        image
        The rules will be processed from top to bottom.
Rule name Function Notes
Redirect root Redirects the root folder to /owa
Exempt OAB from SSL Turns off the requirement for SSL for the /OAB subfolder.
This is the default Exchange 2007/2010 setting. If you want to have SSL for the OAB folder as well, delete this rule and update the OAB URL setting in Exchange.		 Stop processing enabled
Exempt PowerShell vdir from SSL This rule is only required for Exchange 2010, but can safely be imported on Exchange 2007. Stop processing enabled
Force HTTPS Enforces HTTPS for all requests to the site.
      • Perform a test

All requests for the root folder or for the /owa folder missing SSL will now be redirected to the OWA logon page. All other subfolders are not redirected, except to enforce SSL, and can be accessed directly.

Note: The Exchange 2007/2010 web folders usually inherit their SSL settings from the Default Web Site, so when you turn off the SSL requirement for the site you also turn it off for the web folders. If, for some reason, any of the subfolders manage the SSL setting in their own context (ie. they do not inherit the SSL setting from the site level) you have to disable Require SSL for those folders as well. If not, the URL Rewrite will not kick in and you will get an error instead of a redirect. The folders associated with Exchange 2007/2010 are:

      • Autodiscover
      • EWS
      • ecp
      • Exchange
      • Exchweb
      • Microsoft-Server-ActiveSync
      • OAB
      • Owa
      • Public
      • Rpc
      • RpcWithCert
      • UnifiedMessaging

All these should have their Require SSL Setting turned off. For any other folders you may have on the server you will have to decide for yourself if you want them to be included in the URL Rewrite SSL Enforce or manage their SSL settings individually. Also make sure to check any other folders that should have SSL active still has that setting set when you deactivate the requirement for the site.

I find this solution to simplifying the URL for OWA to be much more streamlined and elegant than any previous solution. The URL Rewrite filer is a module developed by Microsoft meaning it has been through the Secure Development Lifecycle (SDL). You have only one place to make all changes. You do not have to make changes to the default IIS configuration (ie. editing or changing the error pages).

Morgan

Update: When you disable the Require SSL setting in IIS you rely on URL Rewrite to perform the enforcement for SSL on your sites and directories. I have been in contact with the author of URL Rewrite, asking him if this configuration is a security risk and if the enforcement of SSL through URL Rewrite is as strong as the one in IIS. His reply was that it was not but that this was a cause for concern in very few situations. You have been warned.

Exchange Server

Remember the HOSTS file!

2 Comments

After moving a virtual Exchange 2007 machine from Virtual Server to Hyper-V I started getting error messages both in the Event Logs and in the Exchange Management Console and Shell. During the move I had also changed the IP address of the server. Here are the error messages:

In the Application Log:

EventID: 4001
Source : MSExchange System Attendant Mailbox
EntryType : Error

Message: A transient failure has occurred. The problem may resolve itself in awhile. The service will retry in 56 seconds. Diagnostic information:

Cannot open mailbox /o=Lab Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=LAB-EXCH/cn=Microsoft System Attendant.

Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException: Cannot open mailbox /o=Lab Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Configuration/cn=Servers/cn=LAB-EXCH/cn=Microsoft System Attendant. —> Microsoft.Mapi.MapiExceptionNetworkError: MapiExceptionNetworkError: Unable to make co

nnection to the server. (hr=0x80004005, ec=2423)

Diagnostic context:

……
Lid: 12952dwParam: 0x6BAMsg: EEInfo: prm[0]: Long val: 135
Lid: 15000dwParam: 0x6BAMsg: EEInfo: prm[1]: Pointer val: 0x00000000
Lid: 15000dwParam: 0x6BAMsg: EEInfo: prm[2]: Pointer val: 0x00000000
Lid: 16280dwParam: 0x6BAMsg: EEInfo: ComputerName: n/a
Lid: 8600 dwParam: 0x6BAMsg: EEInfo: ProcessID: 2372
Lid: 12696dwParam: 0x6BAMsg: EEInfo: Generation Time: 2009-03-26 13:01:49:138
Lid: 10648dwParam: 0x6BAMsg: EEInfo: Generating component: 8
Lid: 14744dwParam: 0x6BAMsg: EEInfo: Status: 10060
Lid: 9624 dwParam: 0x6BAMsg: EEInfo: Detection location: 318
Lid: 13720dwParam: 0x6BAMsg: EEInfo: Flags: 0
Lid: 11672dwParam: 0x6BAMsg: EEInfo: NumberOfParameters: 0
Lid: 19778
Lid: 27970StoreEc: 0x977
Lid: 17730
Lid: 25922StoreEc: 0x977

at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, Int32 ec, DiagnosticContext diagCtx)

at Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, Object objLastErrorInfo)

at Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize)at Microsoft.Mapi.ConnectionCache.OpenMapiStore(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, ClientIdentityInfo clientIdentity, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)at Microsoft.Mapi.ConnectionCache.OpenMailbox(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, WindowsIdentity windowsIdentityAs, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)

— End of inner exception stack trace —

at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)at Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)at Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags)at Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString)at Microsoft.Exchange.Data.Storage.MailboxSession.OpenAsAdmin(ExchangePrincipal mailboxOwner, CultureInfo cultureInfo, String clientInfoString, Boolean useLocalRpc, Boolean ignoreHomeMdb)at Microsoft.Exchange.Data.Storage.MailboxSession.OpenAsAdmin(ExchangePrincipal mailboxOwner, CultureInfo cultureInfo, String clientInfoString)at Microsoft.Exchange.Servicelets.SystemAttendantMailbox.Servicelet.Work()

In Exchange Management Console (EMC) and Exchange Management Shell (EMS) when viewing the Mailbox node under Server Configuration:

——————————————————–

Microsoft Exchange Warning

——————————————————–

The following warning(s) were reported while loading topology information:

get-MailboxDatabase

Completed

Warning:

ERROR: Could not connect to the Microsoft Exchange Information Store service on server <server FQDN>. One of the following problems may be occurring: 1- The Microsoft Exchange Information Store service is not running. 2- There is no network connectivity to server lab-exch.lab.net. 3- You do not have sufficient permissions to perform this command. The following permissions are required to perform this command: Exchange View-Only Administrator and local administrators group for the target server. 4- Credentials have been cached for an unpriviledged user. Try removing the entry for this server from Stored User Names and Passwords.

This last error occurs when either the Management Console or Management Shell executes the Get-MailboxDatabase -Status cmdlet.

Outlook Web Access also did not work and displayed this error:

Outlook Web Access could not connect to Microsoft Exchange. If the problem continues, contact technical support for your organization.

Request

Url: https://<servername>:443/owa/default.aspx

User host address: <address of client>

Exception

Exception type: Microsoft.Exchange.Data.Storage.ConnectionFailedTransientException

Exception message: Cannot open mailbox /o=Lab Organization/ou=Exchange Administrative Group (FYDIBOHF23SPDLT)/cn=Recipients/cn=morgan.

Call stack

Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)

Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)

Microsoft.Exchange.Data.Storage.MailboxSession.Initialize(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags)

Microsoft.Exchange.Data.Storage.MailboxSession.CreateMailboxSession(LogonType logonType, ExchangePrincipal owner, DelegateLogonUser delegateUser, Object identity, OpenMailboxSessionFlags flags, CultureInfo cultureInfo, String clientInfoString)

Microsoft.Exchange.Data.Storage.MailboxSession.Open(ExchangePrincipal mailboxOwner, WindowsPrincipal authenticatedUser, CultureInfo cultureInfo, String clientInfoString)

Microsoft.Exchange.Clients.Owa.Core.OwaWindowsIdentity.CreateMailboxSession(ExchangePrincipal exchangePrincipal, CultureInfo cultureInfo)

Microsoft.Exchange.Clients.Owa.Core.UserContext.CreateMailboxSession(OwaContext owaContext)

Microsoft.Exchange.Clients.Owa.Core.UserContext.Load(OwaContext owaContext)

Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.CreateUserContext(OwaContext owaContext, UserContextKey userContextKey, UserContext& userContext)

Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.PrepareRequestWithoutSession(OwaContext owaContext, UserContextCookie userContextCookie)

Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.InternalDispatchRequest(OwaContext owaContext)

Microsoft.Exchange.Clients.Owa.Core.RequestDispatcher.DispatchRequest(OwaContext owaContext)

Microsoft.Exchange.Clients.Owa.Core.OwaModule.OnPostAuthorizeRequest(Object sender, EventArgs e)

System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()

System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)

Inner Exception

Exception type: Microsoft.Mapi.MapiExceptionNetworkError

Exception message: MapiExceptionNetworkError: Unable to make connection to the server. (hr=0x80004005, ec=2423) Diagnostic context: …… Lid: 12952 dwParam: 0x6BA Msg: EEInfo: prm[0]: Long val: 135 Lid: 15000 dwParam: 0x6BA Msg: EEInfo: prm[1]: Pointer val: 0x00000000 Lid: 15000 dwParam: 0x6BA Msg: EEInfo: prm[2]: Pointer val: 0x00000000 Lid: 16280 dwParam: 0x6BA Msg: EEInfo: ComputerName: n/a Lid: 8600 dwParam: 0x6BA Msg: EEInfo: ProcessID: 3372 Lid: 12696 dwParam: 0x6BA Msg: EEInfo: Generation Time: 2009-03-25 20:59:22:652 Lid: 10648 dwParam: 0x6BA Msg: EEInfo: Generating component: 8 Lid: 14744 dwParam: 0x6BA Msg: EEInfo: Status: 10060 Lid: 9624 dwParam: 0x6BA Msg: EEInfo: Detection location: 318 Lid: 13720 dwParam: 0x6BA Msg: EEInfo: Flags: 0 Lid: 11672 dwParam: 0x6BA Msg: EEInfo: NumberOfParameters: 0 Lid: 19778 Lid: 27970 StoreEc: 0x977 Lid: 17730 Lid: 25922 StoreEc: 0x977

Call stack

Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, Int32 ec, DiagnosticContext diagCtx)

Microsoft.Mapi.MapiExceptionHelper.ThrowIfError(String message, Int32 hresult, Object objLastErrorInfo)

Microsoft.Mapi.ExRpcConnection.Create(ConnectionCache connectionCache, ExRpcConnectionCreateFlag createFlags, ConnectFlag connectFlags, String serverDn, String userDn, String user, String domain, String password, String httpProxyServerName, Int32 ulConMod, Int32 lcidString, Int32 lcidSort, Int32 cpid, Int32 cReconnectIntervalInMins, Int32 cbRpcBufferSize, Int32 cbAuxBufferSize)

Microsoft.Mapi.ConnectionCache.OpenMapiStore(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, ClientIdentityInfo clientIdentity, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)

Microsoft.Mapi.ConnectionCache.OpenMailbox(String mailboxDn, Guid mailboxGuid, Guid mdbGuid, WindowsIdentity windowsIdentityAs, String userDnAs, OpenStoreFlag openStoreFlags, CultureInfo cultureInfo, String applicationId)

Microsoft.Exchange.Data.Storage.ConnectionCachePool.OpenMailbox(String serverDn, String userDn, String mailboxDn, Guid mailboxGuid, Guid mdbGuid, Object identity, ConnectFlag connectFlag, OpenStoreFlag openStoreFlag, CultureInfo cultureInfo, String clientInfoString, Boolean secondTry)

The Exchange Best Practices Analyzer (BPA) also failed when run locally against the server. The error was: Exchange Server cannot be contacted.

Despite these errors all Exchange services were running and Outlook clients on the internal network could connect to Exchange and open their mailbox. Any access through CAS did not work, so Outlook Anywhere or OWA were unavailable.

After searching the web I did not find an exact match for these errors, at least none that had my particular symptoms. But from what I read it looked like an issue with the network configuration, i.e. the network card.

First I went into Device Manager and removed all missing hardware devices, this included the old Intel 21140 emulated NIC from Virtual Server. I had not removed the Virtual Server Virtual Machine Additions until after I had moved the server so its configuration was still in the Registry. For information on how to remove missing devices, have a look here. Finally I removed the Virtual Machine Bus Network Adapter, reinstalled it and set the correct IP address information. All this did not resolve the error either, but I discovered something interesting in the Registry regarding the Virtual Machine Bus Network Adapter, which I will cover in a later post.

Still working with the networking idea I ran the Exchange Best Practices Analyzer (BPA) from a different server, and this time I could connect and complete the scan. This meant that at least the network was working and that the Windows Firewall was not blocking the communication. The next step was to run Wireshark on the server while I ran the BPA against the local server, that is not over the network. This produced the first real indication on what the error was. Here is a screenshot from the capture:

032609_1821_Rememberthe1

The Exchange server’s new IP address on Hyper-V was 192.168.131.3. But while it was hosted on Virtual Server it’s IP address was 192.168.131.180. So judging from the ARP broadcasts, which are coming from the local server, the old IP address is still active somehow. To test this theory I changed the server’s IP address back to the old one 192.168.131.180. Now the BPA finished successfully and EMC/EMS were able to successfully run the Get-MailboxDatabase -Status cmdlet. I changed the IP address back to the new one, 192.168.131.3, hoping that this would somehow reset the erroneous configuration. It did not, and the errors immediately came back. So the old IP was still somewhere in the Registry. I did a search in the Registry for 192.168.131.180, and this is what I got:

reg query HKLM /s /f “192.168.131.180”
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionNetworkListNlaCacheIntranetAuth2.4.fe80::7de2:4df0:fa6e:2231%10 1.1.192.168.131.180
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionNetworkListNlaCacheIntranetAuth2.4.fe80::d0b2:3082:e67:d254%10 1.1.192.168.131.180
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionNetworkListNlaCacheIntranetAuth2.4.fe80::e1e0:a13d:50cc:f9b3%10 1.1.192.168.131.180
HKEY_LOCAL_MACHINESYSTEMControlSet001Services{3832C4EC-AC88-42FA-B266-51D68342079A}ParametersTcpip
IPAddress REG_MULTI_SZ 192.168.131.180
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServices{3832C4EC-AC88-42FA-B266-51D68342079A}ParametersTcpip
IPAddress REG_MULTI_SZ 192.168.131.180
End of search: 5 match(es) found.

I ignored everything under the NLA key which is part of the Network Location Awareness service. That left the GUID key ({3832C4EC-AC88-42FA-B266-51D68342079A}) under Services. Each GUID key under the Services key represents a Network Card. So which card was this?

Windows maintains a list of the network cards in the system under the Registry key HKLMSOFTWAREMicrosoftWindows NTCurrentVersionNetworkCards. This is the list from my system:

reg query “HKLMSOFTWAREMicrosoftWindows NTCurrentVersionNetworkCards” /s
HKEY_LOCAL_MACHINESOFTWAREMicrosoftWindows NTCurrentVersionNetworkCards8
ServiceName REG_SZ {3832C4EC-AC88-42FA-B266-51D68342079A}
Description REG_SZ Microsoft Virtual Machine Bus Network Adapter

So this was the Virtual Machine Bus Network Adapter. Time to examine the TCP/IP registry settings for this adapter more closely.

The main TCP/IP configuration location in the Registry is under HKLMSYSTEMCurrentControlSetServicesTcpip. The configuration of the network cards is in the subkey ParametersInterfaces:

reg query HKLMSYSTEMCurrentControlSetServicesTcpipParametersInterfaces /s
HKEY_LOCAL_MACHINESYSTEMCurrentControlSetServicesTcpipParametersInterfaces{3832C4EC-AC88-42FA-B266-51D68342079A}
UseZeroBroadcast REG_DWORD 0x0
EnableDeadGWDetect REG_DWORD 0x1
EnableDHCP REG_DWORD 0x0
NameServer REG_SZ 192.168.131.2
Domain REG_SZ
RegistrationEnabled REG_DWORD 0x1
RegisterAdapterName REG_DWORD 0x0
DhcpServer REG_SZ 255.255.255.255
Lease REG_DWORD 0x1c20
LeaseObtainedTime REG_DWORD 0x49caa3c7
T1 REG_DWORD 0x49cab1d7
T2 REG_DWORD 0x49cabc63
LeaseTerminatesTime REG_DWORD 0x49cabfe7
AddressType REG_DWORD 0x0
IsServerNapAware REG_DWORD 0x0
DhcpConnForceBroadcastFlag REG_DWORD 0x1
IPAddress REG_MULTI_SZ 192.168.131.3
SubnetMask REG_MULTI_SZ 255.255.255.0
DefaultGateway REG_MULTI_SZ 192.168.131.1
DefaultGatewayMetric REG_MULTI_SZ 0

As we can see there is a configuration mismatch between the configuration under the GUID key ({3832C4EC-AC88-42FA-B266-51D68342079A}) under Services, and the configuration under the Tcpip key. Seemed like a buggy TCP/IP stack so I decided to reset it:

  1. Uninstalled the Virtual Machine Bus Network card from Device Manager.
  2. Reset WinSock:
    netsh winsock reset catalog
  3. Reset TCP/IP:
    netsh int ip reset %temp%reset.log
  4. Reboot

These steps would completely reinstall TCP/IP and reinstall the NIC.

But after the reboot the errors were still present and Wireshark saw the same ARP requests for the old address 192.168.131.180. The old address was now completely gone from the Registry and Active Directory does not store IP addresses (usually). So where was this information coming from? I looked in the forward and reverse lookup zones on the DNS server, but the correct IP was present in the forward zone. The old address was in the reverse zone, but that was not relevant here. I did a ping from the command line for the server name, and it returned the old address! But Nslookup did not! Finally I was on to something! The HOSTS file!

Here is the hosts file on the server:

# Copyright (c) 1993-2006 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a ‘#’ symbol.
#
# For example:
#
# 102.54.94.97 rhino.acme.com # source server
# 38.25.63.10 x.acme.com # x client host
127.0.0.1 localhost
#::1 localhost
192.168.131.180    lab-exch
192.168.131.180    lab-exch.lab.net

And there was the answer and the explanation for everything. I had previously entered the information for the local server into the hosts file to work around a bug in the DSProxy components on the CAS server where DSProxy would not listen on IPv6 addresses. More info on this issue here. This also explained why other machines could connect to the server and why EXBPA failed when run locally.

I updated the addresses and rebooted. The issue with DSProxy and IPv6 was fixed in Update Rollup 4 for Exchange 2007 with Service Pack 1, but I was not able to make it work in my setup without the entries in the hosts file. My guess is that this is because I have an ISA server between the Exchange server and the clients.

So there you have it. A very simple solution arrived at via very long, cumbersome and unnecessary process. So remember that old hosts file guys.

Exchange Server

Troubleshooting Outlook Anywhere (Outlook RPC-over-HTTPS)

1 Comment

This is an unordered list of tools and functionality that is handy when troubleshooting Outlook Anywhere or Outlook RPC-over-HTTPS as it was previously known.

  • Wireshark
    Excellent network protocol analyzer, use it to see what Outlook and Exchange send over the wire. It is possible to configure Wireshark to decrypt an SSL stream, I will post a how-to about this soon.
  • Outlook Debug Logging
    Enable from ToolsàOptionsàOtheràAdvanced Options…àEnable logging (troubleshooting)
    This will generate log files in the user’s temp folder as well as events in the Event Log.
  • Outlook Connection Status/Test E-Mail AutoConfiguration…
    Access by right clicking the Outlook icon in the system tray/notification area while pressing CTRL.
    In the Connection Status window you can see server names, connection types, network interface, protocols, status and statistics. Also you can view the activity in your local mailbox. The Test E-Mail AutoConfiguration option shows you all the information delivered by the Autodiscover web service.
  • Outlook RPC diag
    Start Outlook with the command line parameter /rpcdiag, eg. outlook.exe /rpcdiag. This brings up the Connection Status box.
  • IIS Log files
    Usually located in your inetpub directory these files show you the requests that you web server receives, as well as the return codes the server responded with.
  • Windows Event Logs
    Look here for errors from the web server or Outlook, or any other component associated with Outlook Anywhere functionality.
  • Test-OutlookWebServices cmdlet
    This cmdlet runs a test on the Autodiscovery web service and outputs the results.

Morgan