Exchange, SharePoint Services and ActiveSync Administration…

Running Exchange 2003 and SharePoint Services on the same machine can be a challenge, as is well documented:
I recently tried to accomplish this, but without much success.
First some background. I was trying to run Exchange 2003 Standard Edition with OWA/OMA and EAS and SharePoint Services 2.0 on the same server, which was also a Domain Controller. Actually, it was the only server in the network.
Things started out great, I installed Exchange and performed the steps outlines in KB817379 (http://support.microsoft.com/kb/817379/), which I had to do to get OMA and EAS to work since I did not have a Front-End Exchange server. After testing all the Exchange services I proceeded with installing SharePoint Services. It installed without incident, but then the trouble started. OWA was not working, the same for OMA and EAS. WSS, however, was working. I started troubleshooting and soon found the article I linked to at the beginning of this post. Following the steps in it I was able to get OWA working, but not OMA and EAS. Every time I tried to sync a phone or access OMA I got an error in the event log that said HTTP 400 Bad request. Bad request means a malformed request was sent to the server.
Literally, for days, I search the net and finally had to put in a call to Microsoft PSS. The technician and I spent even more days looking at logfiles and traces of communication between server and clients. We were stomped. I was seriously beginning to contemplate reinstalling the server from scratch. But then we found something.
If you read article KB817379, in step 21 you see the value of the new VDir registry key you will create. Not that the article explicitly states that it should be /exchange-oma, with a forward slash. When we looked at the trace logs from Exchange we found this:
As you can see we had double forward slashes. This was the cause of the Bad Request message. An HTTP GET requestcontaining two slahes is obviously malformed. When I removed the slash from the VDir value in the registry and restarted IIS all services worked.
When this problem was resovled I installed the ActiveSync Mobile Administrator. That was also pretty hard to get working together with WSS, but I succeded.

Returning Search Statistics from Active Directory

When you search Active Directory you have the option of asking the server to return search statistics for your query. This is done by adding an LDAP control to your query. The control is 1.2.840.113556.1.4.970. I was using LDP to test this out on my test forest. The forest runs Windows Server 2003 DCs and I use the W2K3 version of the Active Directory Administration Tool; LDP. In that version of LDP it is very easy to add the required LDAP control to the search, you just select it from a drop-down box from the Search Controls dialog.

I specifically wanted to see the expanded queries that the server extrapolates when using ANR searches, or LDAP Display Names in objectCategory searches. But to my surprise, the server never returned any statistics. I always got the same standard output; the result of my search and nothing more. After a little investigation I found the following text on the MSDN site:

To retrieve all of the above information [search statistics], the account that issues the LDAP request should have debug privileges in its token.

I remembered having tested PWDUMP on that particular server. PWDUMP requires the debug privilege to work. I had removed that from the Administrators group to test something with PWDUMP. When I gave it back and logged off and on again i could successfully see the search statistics from the domain controller.

Nice!

Linked attributes in Active Directory

In Active Directory there is something called linked attributes. They exist in pairs, consisting of a forward-link and a back-link. The linked attribute pair member, of Group objects, and memberOf, of User or Groups is an example. In this particular case member is the forward-link and memberOf is the back-link. Back-links are always calculated automatically by the system whenever an attribute that is a forward-link is modified. If you change the member attribute of a group and add another object, the groups DN is automatically added to the memberOf attribute of the object you added.

I wanted to find out a little more about how this worked so I created a couple of scripts to do some testing. Specifically I wanted to see if I could write directly to a back-link attribute.

The first script tried to do that. It connected to an object in the directory and tried to write the DN of a group into the memberOf attribute. That failed with the error:

Code: 80072035
Error: The server is unwilling to process the request.

The next script connected to a group and added a user to it. As expected, that worked. I examined the user I added to the group in ADSI Edit and the back-link memberOf had been correctly computed. From that I can draw the conclusion that the computing of back-links is implemented in the DSA itself and not in the Admin tools (I was using a script, not ADUC, remember).

Next, I tried to edit the memberOf attribute of the user I had just added to the group directly in ADSI Edit. That provided the last piece of the puzzle and a conclusive answer to my question. Because that failed with the following error:

Access to the attribute is not permitted because the attribute is owned by the Security Accounts Manager (SAM).

So that was it. You cannot write to the back-link of a linked attribute pair. The back link is always automatically calculated and added by the system.

Also, the calculation and updating of a back-link attribute does not qualify as a change of the object. When I added a user to a group, only the group’s whenChanged attribute was updated. The user’s remained unchanged. That means that it is always the group object that is modified when you add a user to it. This seems obvious, but consider that you can also do this from the user’s properties on the Member Of tab. What you are actually doing is editing the group object, not the user.

Ain’t Active Directory fun!

Creating users with passwords that do not meet complexity requirements

I am currently reading John Craddock and Salley Storey’s book Active Directory Forestry. It is a very good read, and I can recommend it highly.

Anyway, I discovered something interesting. The book mentions some attributes of the RID Set object of Domain Controllers, specifically the rIDNextRID attribute of that object. rIDNextRID holds the value of the RID that will be given to the next object created on that domain controller. On my DC that value was 1130. Just for fun I created a new user and checked the value of rIDNextRID, only to discover that it had jumped two values insted of one, to 1132. This was a lone DC and no other objects had been created at the same time, so that was not the reason. I checked the SID of the newly created user and discovered that it had a RID of 1131. So which object had the 1130 RID?

I searched the forest for that SID and the search returned 0 objects. But then I asked LDP to return deleted objects as well, and then I got a match. The object with RID 1130 had been deleted at nearly exactly the same time that my test user had been created. The deleted object also had been named exactly the same as my test user. What had happened?

The answer was that when I created my test user I specified a password that did not meet the password requirements of my domain. You complete the create user wizard completely before you get that error from the directory service. So what happened was that the user had been created in the directory, but had been immediatley deleted by the system when it was found that the password did not meet the policy.

In the user interface I could simply click the back button and reenter a new password. That however triggered the creation of a new user, identical to the first one, but now with a compliant password.

I found this to be an interesting quirk and a nice insight into how the administative tools work.

Finding free space in Exchange Server 2003 Databases

Finding out how much free space (white space) an Exchange database has is something I regularly do. I want to show you two ways you can get this information.

First of, you can use ESEUTIL /MS <database name> to dump the free space information, but this requires that the database is dismounted. That is not always practical, since users may want to read some email while you check for free space.

The second way is to view the Application log of the Exchange server. When Exchange finishes online defragmentation, which is scheduled to run each night by default, it logs an event with ID 1221 in the Application log. The source can either be MSExchangeIS Mailbox Store or MSExchangeIS Public Store, dependig on whether the store is a mailbox or a public folder store. A typical 1221 event looks like this:

Type:         Information
Event:        1221
Date Time:    11.07.2006 06:00:00
Source:       MSExchangeIS Mailbox Store
ComputerName: SERVER1
Category:     General
User:         N/A
Description:  The database “Storage Group Astore1” has 659 megabytes of free space after online defragmentation has terminated.

Another neat trick is to use the eventquery.vbs tool to dump all the recent 1221 events so you can easily see all your databases. A sample commmand to dumpt the events from the last 24 hrs would be:

cscript c:windowssystem32eventquery.vbs /L Application /V /FI “ID eq 1221” /FI “DATETIME eq 07/11/06,01:00:01AM-07/11/06,11:59:59PM” /FO LIST

Outlook tip

Just discovered a new hotkey in Outlook that I previously did not know about. When typing names in the To field you are often presented with the drop down list of previously resolved addresses from the Outlook local cache. This cache is stored in a file usually called outlook.nk2 in you userprofile directory tree. But if you have not previously used the address and want Outlook to try to resolve it you can press CTRL+K. This is the same as pressing the Check Names button on the toolbar.

Difference betwen groups in the Builtin container and Domain Local groups

Group sAMAccountType groupType systemFlags isCriticalSystemObject
——————————————————————————
Built-in 536870912 -2147483643 -1946157056 Yes
DL       536870912 -2147483644 <Not Set> <Not Set>The groups in the Builtin container may look like ordinary Domain Local groups, but they are not. In Windows Server 2003 Active Directory they are listed as Builtin Local.
These groups cannot be used on other machines in a domain when the domain is in native mode (as can other Domain Local groups in native mode).
The builtin groups are only valid on Domain Controllers.

<— From ADSI SDK : Start ————————————————————————>
The groupType attribute in Active Directory contains the group type definition:
ADS_GROUP_TYPE_GLOBAL_GROUP
ADS_GROUP_TYPE_DOMAIN_LOCAL_GROUP
ADS_GROUP_TYPE_UNIVERSAL_GROUP
ADS_GROUP_TYPE_SECURITY_ENABLED

The first three flags specify the group scope.
The ADS_GROUP_TYPE_SECURITY_ENABLED flag indicates the type of the group. If this flag is set, the group is a security group. If this flag is not set, the group is a distribution group.ADS_GROUP_TYPE_GLOBAL_GROUP        = 0x00000002
ADS_GROUP_TYPE_DOMAIN_LOCAL_GROUP  = 0x00000004
ADS_GROUP_TYPE_LOCAL_GROUP         = 0x00000004
ADS_GROUP_TYPE_UNIVERSAL_GROUP     = 0x00000008
ADS_GROUP_TYPE_SECURITY_ENABLED    = 0x80000000

ADS_GROUP_TYPE_GLOBAL_GROUP
Group that contains only accounts and other account groups from its own domain.
This group may be exported to a different domain.

ADS_GROUP_TYPE_DOMAIN_LOCAL_GROUP
Group that can contain accounts and universal groups from any domains. It may not be included in either access-control lists of resources in other domains or groups other than global groups in the same domain.

ADS_GROUP_TYPE_LOCAL_GROUP
This flag is for the WinNT provider as the
ADS_GROUP_TYPE_DOMAIN_LOCAL_GROUP flag is for the LDAP provider.

ADS_GROUP_TYPE_UNIVERSAL_GROUP
Group that can contain accounts and account groups from any domains, but not domain local groups.

ADS_GROUP_TYPE_SECURITY_ENABLED
Group that is security enabled. This group can be used to apply an access-control list on an ADSI object or a file system.
<— From ADSI SDK : End ————————————————————————–>

In reality there exists a fifth group type:
ADS_GROUP_TYPE_BUILTIN_LOCAL_GROUP = 0x00000001

These groups can only be security enabled, and when they do, they have a groupType attribute with the value -2147483643.

Exchange group relationships in a multi-domain forest

In a multi-domain forest with Exchange 2000/2003 installed there are some special group relationships.
Each domain for which DomainPrep has been run, has the following Exchange related groups:
  • Exchange Domain Servers (Global Group)
  • Exchange Enterprise Servers (Domain Local Group)

Exchange Enterprise Servers
Purpose: Group all Exchange servers in a specific Enterprise (organization/forest)
This group has the follwing members:
– The computer account of all Exchange servers in the organization
– The Exchange Domain Servers group from all domains where DomainPrep has been run

Exchange Domain Servers
Purpose: Group all Exchange Servers in a specified domain
This group has the following members:
– The computer account of all Exchange servers in the domain where the group exists

Errors
When adding a new Recipient Update Serveice to a domain in a multi-domain forest, that previously has not had Exchange, it is quite usual to get the following errors in the application log on the Exchange server (Exchange server is usually located in another domain):

Source:  MSExchangeAL
Category: LDAP Operations
Event ID: 8270
Description: LDAP returned the error [32] Insufficient Rights when importing the transaction
dn: <GUID=A907D19B-18F7-4098-95AB-A8E029C1634C>
changetype: Modify
member:add:<GUID=E480D07A-1A37-4D43-BC52-9A59958F3DD9>

In this event the dn: <GUID> is the GUID of the ‘Exchange Enterprise Servers’ group in the domain specified in the event. The member:add:<GUID> is the GUID of the ‘Exchange Domain servers group’ from another domain.
Probably a domain that was recently added to the forest. You will see this error for each of the other domains in the forest. The event will be repeated but with a different GUID in the member:add field.

You will also see this error:

Source:  MSExchangeAL
Category: LDAP Operations
Event ID: 8270
Description: LDAP returned the error [32] Insufficient Rights when importing the transaction
dn: <SID=0102000000000005200000002A020000>
changetype: Modify
member:add:<GUID=E480D07A-1A37-4D43-BC52-9A59958F3DD9>

In this event the <SID> is the SID of the ‘Pre-Windows 2000 Compatible Access’ group in the domain that is specified in the event (dc=xxx,dc=xxx),and the member:add:<GUID> is the GUID of the ‘Exchange Domain Servers’ group in one of the other domains in the forest. You will see this error for each of the other domains in the forest. The event will be repeated but with a different GUID in the member:add field.
These errors are most likely due to incorrect permissions in the target domain’s Active Directory.
The permissions are not correctly set or all information in not yet replicated.

Thus we can deduce the follwing member relationships:
Group Name:                          Memebership:
Exchange Domain Servers         All Exchange Servers in the group’s domain
Exchange Enterprise Servers      Exchange Domain Servers from each additional domain in the forest
Pre-Windows 2000 Compatible    Access Exchange Domain Servers from each additional domain in the forest

There is also a KB article that deals with this here:

Missing permissions cause the Recipient Update Service not to process accounts in Exchange 2000 Server and Exchange Server 2003

Logging on through Terminal Services on a Windows Server 2003 Domain Controller

I work extensively with multi-domain forests, usually in a configuration with an empty root domain and several child domains that host users and computers etc. The other day I was trying to log on to a newly added Domain Controller in a child domain. I was going to prepare the domain for Exchange so I was trying to log on with an account that was a member of the Enterprise Admins group in the root domain. This usually works very well, because this group is always a member of the local Administrators group in any child domain. This time however I could not log on and got this error message:
To log on to this remote computer, you must be granted the Allow log on through Terminal Services right. By default, members of the Remote Desktop Users group have this right. If you are not a member of the Remote Desktop Users group or another group that has this right, or if the Remote Desktop Users group does not have this right, you must be granted the right manually.

This, of course, led me to investigate. The first thing I discovered was that this Domain Controller had also been made a Terminal Server. I won’t go into how bad an idea that is here, suffice to say that I do not recommend it. From this fact it followed that someone also probably had changed the default settings of the Allow log on through Terminal Services right in some policy, and that was probably the reason I could not log on. Sure enough. The Default Domain Policy had been changed (again, not a good idea), granting the Allow log on through Terminal Services right to a global group in the domain only. Let’s call that group TSUsers. I also discovered that the same someone had also added the TSUsers group to the Remote Desktop Users group. Normally that should have been enough to allow log on through Terminal Services. Obviously it wasn’t. So I had two problems. First, why could I not log on as a member of the Administrators group when the Default Domain Policy had been changed, and second; why was it not enough to add the TSUsers group to the Remote Desktop Users group to allow them to log on through Terminal Service?
By default, the Allow log on through Terminal Services right is controlled through the Local Computer Policy, the one you can edit with gpedit.msc. The default setting for Windows Server 2003 is to grant this right to the Administrators and Remote Desktop Users local groups. If the server is promoted to a Domain Controller, the Remote Desktop Users group is removed from the Local Computer Policy, leaving only the Administrators group.  So on a Domain Controller it is not enough to be a member of the Remote Desktop Users group to log on through Terminal Services. You must be a member of the Administrators group in the domain. That is probably what confused the person who had set up the server. He had added his domain group (TSUsers) to the Remote Desktop Users group and been unable to log on, since the server was a DC. That answered my second question. To solve this problem he edited the Default Domain Policy and gave the right to his domain group. But in doing so he overrode the Local Computer Policy, which gives members of the Administrators group access. This was what made me unable to log on to the server, and the answer to my first question. Easy!
Interestingly enough the text in the Remote tab on a Domain Controller does not change, even if Remote Desktop Users no longer can log on through Terminal Services. It still says that members of the group has access.
To solve my immediate problem I added the Enterprise Admins group to the Default Domain Policy in the child domain and was able to log on and do my Exchange preparation. This setup is obviously not recommended. A DC should never be a Terminal Server and domain based policies should not be changed in such a way as to lock out administrators.
In researching this post I also found out another interesting thing about Terminal Services in Windows Server 2003. You no longer have to give a user or group both the Log on locally and Allow log on through Terminal Services rights to be able to log on via Terminal Services. This was needed in Windows 2000. In Windows Server 2003 it is handled this way:
  • Log on locally controls logon via the console (not RDP console, but keyboard attached to the server)
  • Allow log on through Terminal Services controls logons via Terminal Services.
You can read more about that in this KB article:
Difference in the user right “Deny log on locally” between Windows 2000 and Windows 2003
http://support.microsoft.com/kb/837954/en-us

How to remove Event Logs from Event Viewer

The Event Viewer management console has several categories depending on the roles of a server. All systems running Windows have Application, Security and System logs, or categories. These logs are represented by .evt files on disk, typically located in the %SystemRoot%system32config directory.

When a system loses a role, eg. it is demoted from Domain Controller to member server, the logs associated with that role remain in the Event Viewer console on that system. This can be quite annoying, not to mention misleading. Not only is the category retained in Event Viewer, but all the events are there as well.

I will not show you how you can remove these logs. The problem is that the .evt files are locked and cannot be deteled. They are locked by the the Event Log Service which cannot be stopped. The solution is to use Mark Russinovich’s excellent utilities PendMoves and MoveFile. You can find them here:

http://www.sysinternals.com/Utilities/pendmoves.html

Windows often needs to replace a file that is in use. This presents a problem when the process using the file cannot be stopped. To resolve this problem Windows has a spesial API that can tell Session Manager to delete that file, or replace it, on the next reboot. The MoveFile utility does just that. It tells Session Manager where to move, or delete, a file on the next reboot, before the system starts it’s services and applications. This info is stored in the registry key HKLMSystemCurrentControlSetControlSession ManagerPendingFileRenameOperations. You can write to this key using WMI or your own app, but I use Mark’s tool since it is already there.

To get rid of eg. the old File Replication Service Log from a server you would first go into Event Viewer and get the path to the .evt file by selecting properties on the log. Usually you would get C:WINDOWSSystem32configNtFrs.evt

Next, run the following command from the directory where you extracted PendMoves and MoveFile:

MoveFile.exe C:WINDOWSSystem32configNtFrs.evt “”

The “” indicates a NULL destination and is interpreted by Session Manager as a delete operation.

Now you can run PendMoves to get a list of any file move/delete operations scheduled for the next reboot.

But to get completely rid of the log we also will have to remove some setting in the registry, or else the Event Log Service would just recreate the file we deleted. The new file would be empty, of course, but the log would remain in Event Viewer.

Continuing the example with the File Replication Service Log, navigate to the key HKLMSYSTEMCurrentControlSerServicesEventlog
This is the main key for the Event Log Service and it has a subkey for each log that Event Viewer displays. Delete the entire key of the log you want to get rid of.

After the next reboot the logs should be gone from Event Viewer.

Sometimes they are not however. This usually happens because the service that uses the log is still set to Automatic startup. For example; when a DC is demoted to member server, the FRS service is not stopped and disabled. If this is the case the registry key you deleted will be recreated by FRS at startup and a new logfile created. So make sure to check all corresponding services before rebooting.