How to Install Cumulative Update (CU) for a 2 Nodes DAG (Exchange 2013/2016)


I notice this last few weeks, a lot of Exchange admins are searching & Requesting information how to install CU on Production Server. First reason, for this High quantity of request is mainly because customer is plaining or are in Hybrid Mode. While your are running a Exchange with HCW, Microsoft recommend to have at least the last CU or the one before installed. So customer need now a good procedure to safely install CU on their production Exchange Servers.

So please find a quick guide\Procedure I have created to assist you in this CU installation.

PS: all my scripts are tested on my lab so please test the script before applying to production.

The procedure is:

  1. Requirements
    1. Prepare AD for Schema Update – Link
    2. Please be aware of this second way of check for Schema update – Link

  2. Patch Node1:
    1. Update the HLB configuration to redirect traffic to Node2
    2. Put Node 1 in Maintenance mode
      1. Use the attached Script – Link
    3. Install Cumulative Update
    4. Reboot the Exchange Server
    5. Remove Node 1 from Maintenance Mode
    6. Run validation Tests on Node1
      1. Check OWA
      2. Check Outlook connectivity
      3. Check Mail flow
  3. Patch Nod2:
    1. Update the HLB configuration to redirect traffic to Node1
    2. Put Node 2 in Maintenance mode
      1. Use the attached Script – Link
    3. Install Cumulative Update
    4. Reboot the Exchange Server
    5. Remove Node 2 from Maintenance Mode
    6. Run validation Tests on Node1
      1. Check OWA
      2. Check outlook connectivity
      3. Check Mail flow

I hope this will help.


Feedback are welcome

How to check Exchange Attribut change using Repadmin while install Exchange CU



I have created a procedure (with a Powershell script – Link ) to install a CU on your exchange server. I have notice in complexed AD environment , the powershell script may fail.

So please find a second way using a old tools Repadmin:

  1. repadmin /showattr * “CN=Microsoft Exchange,CN=Services,CN=Configuration,dc=Company,dc=com” /atts:objectVersion
  2. repadmin /showattr * “CN=ms-Exch-Schema-Version-Pt, CN=Schema, CN=Configuration,dc=Company,dc=com” /atts:rangeUpper
  3. repadmin /showattr * “CN=Microsoft Exchange System Objects,dc=Company,dc=com” /atts:objectVersion

Which CU is updating the Schema and\or AD:

  1. Exchange 2013:
  2. Exchange 2016 :

How to build an Exchange Hybrid Environment (CheckList)


I have been working this last year on number of Migration using third party tools or Hybrid deployment.

So please find a way to Design & deploy a working Exchange Hybrid Environment.

Requirements & Preparation

Active Directory
Forest & Domain Functional Level:

  • Windows 2008 DC to be able to use writeback (AD Connect)
  • Windows 2012R2 Forest level: Workplace Join (AD FS)
Schema Update:

  • Some Exchange CU required Schema Update
User authentication method:

  • UserPrincipalName (UPN):
  • SAMAccountName:
    • contoso\user
Domain suffix will be used for SSO
The Domain must be registered as a public domain with Public DNS

  • Yes: advise to use UPN
  • No: Add a new Domain Suffix or consider to use AlternateID
UPN must be know set and known by the user

  • UPN is match user email address (Recommendation)
Run Microsoft Office 365 Deployment Readiness Tool (user Impact)
Run IdFix to validate Users Readiness for Office 365 migration
Supported Exchange & Client:

  • Exchange Version:
  • Client Connectivity: Outlook\OWA\EAS\OA (version & enable)
Required Exchange Servers for Hybrid:

  • % of available resources (Exchange Calculator)
  • Additional server is needed
Namespace publicly available: EWS, OWA, Autodiscover, OA, EAS
(Check using Microsoft Remote Connectivity Analyzer)
Accepted Domains
Availability Space or Federation Trust (Free\Busy Sharing)
Autodiscover is pointing to your on-premise Exchange organization

  • Publish External
  • MRS Proxy is enabled
Valid Certificate from Third Party Provider (Exchange Web Services)
Self-Signed certificate for Federation Trust with MFG (only)
EWS External URL specified in your public DNS listed in the SAN
Autodiscover Endpoint specified in your public DNS listed in the SAN
All Exchange Server used for mail transport in Hybrid Deployment must use the same certificate (same CA & same Subject)

  • Hybrid Server to Exchange Online
  • Hybrid Server to EOP (Mail Flow)
Office 365
Sign up for Office 365
Valid Global Admins from your office 365 Tenant
Validation of custom Domain

  • Every accepted domain need to be set as office 365 custom domain
  • UPN suffix need to set as custom domain
  • Validation using TXT Record
  • Transfer DNS Management if no customization is needed
Directory Synchronization
AD Connect
AD Connect: On-Premise AD – Enterprise Admin
AD Connect: On-Premise Service Account for Federation Service
AD Connect: SQL requirements (Local or Existing SQL Server\Instance)
AD Connect : Firewalls

  • AD Connect -> DCs
  • AD Connect -> ADFS & WAP
  • AD Connect -> Azure AD
Method
ADFS (SSO): Request Servers according the design
ADFS (SSO): Request SQL Instance or WID
ADFS (SSO): A record for Federation Service (Internal DNS)
ADFS (SSO): DNS Entry for Federation Service (Public DNS)
ADFS (SSO): On-Premise Service Account for Federation Service
ADFS (SSO): Pfx certificate with federation service name
ADFS (SSO): Local administrator account for WAP Server(s)
ADFS (SSO): Enable Remote Management for WAP Server
ADFS (SSO): Firewalls

  • ADFS Servers DCs
  • WAP ADFS Server
  • WAP Internet users
Identity Management
AD Connect
How many AD Object to sync to office 365?
Synchronization using OU, Group or Domain
User Principal Name:
Select an Attribute (Default Value : UserPrincipalName (UPN)
This Attribute match the User Primary SMTP address
SourceAnchor selection (ObjectGUID Default)
Features needed:

  • Exchange Hybrid
  • Password Synchronization
  • Password writeback
  • Group Writeback (Office 365 Group)
method
Password Synchronization
Federated with ADFS
How many farm do we need?
How many server do we need ? More than 5?
Federation Service
Enable HA
Using SQL Server
Do we need SQL or WID will be enough
Do we need SQL for AD Connect
Do we need SQL for Federation Servers
Hybrid Environment
Certificate Design

  • Mail Transport (secure communication)
  • Autodiscover, OWA, EAS, OA
Web Services:

  • External FQDN of your Hybrid Server
    • Can be the external LB FQDN to reach your servers
    • Can be OWA external FQDN
Client Connectivity (OWA and EAS)
Availability Space or Federation Trust
Federated & Accepted Domain
Mail flow (Centralized mail Transport)
Federation Trust and Organization Relationships
Hybrid Installation & Configuration
AD Connect
AD Connect Installation & Configuration
Use AD Connect to install ADFS Farm (ADFS Server & WAP)
Configure Hybrid Configuration using HCW:

  • Exchange org Admin (On-Premise)
  • Global Admin (office 365)
  • Federation trust validation (TXT Record)
  • Secure Transport Mail Flow
    • Receive Connector
    • Send Connector
    • Certificate for Secure Mail Transport
    • External FQDN of Hybrid Server for EOP
Validation (checks)
AD Connect
Directory Synchronization:

  • Search for SMTP matching if license already assign
Sign in using SSO (Redirect, ADFS authentication)
Free\Busy Sharing:

  • Office 365 users to On-premise users
  • On-premise users to Office 365 Users
Cross-Premise Features:

  • MailTips
  • Message Tracing
  • Multi-mailbox Search
Online Archiving
OWA redirect\ Exchange ActiveSync Redirect
Cross-Premise Mailbox Permission (Conditions applied)
Unified GAL
Migration (Onboarding)

Operation & Migration (Onboarding)

How to create an User\Mailbox using EAC from on-premise Exchange
Migration Procedure:

  • Using Powershell Script (Sync, Licenses, Complete)
  • Using third Party application
Objects to Migrate:

  • Distribution list
  • Contacts
  • User Mailbox
  • Resource Mailbox
  • Shared Mailbox


Known issues Cross-Premise Mailbox Permission

  • Send as
  • Send on behalf
  • Full Access:
    • Already create (will be migrated)
    • Add full permission
Cross-premise Free\Busy

Comment & question are welcome so please do not hesitate.



PS1 – Maintenance mode for Exchange 2013 & 2016


Please find a nice script I have wrote to assist with the installation of Cumulative update on Exchange 2013/2016.

The script is following the both Technet procedures:

  • Exchange 2013: Link
  • Exchange 2016: Link

Please do not hesistate to let me know what you think about the script?

Script:Exchange 2013 Maintenance Mode

PS: Please test the script before applying to your environment.the script is provide as it is.

Autodiscover within Office 365 Hybrid (Design & Recommandations)

While providing Support to Office 365 customers (mostly hybrid), I have noticed most frequent reason of issue in Hybrid environment come from the following reasons:

  1. Hybrid Server Design
  2. Autodiscover Design

In Hybrid Scenario, the Design of  Hybrid Server is exactly the same as on-premise Internet facing Client Access Server. you have to follow some basic Exchange Recommandations.

  • Your Most Recent Version of Exchange Server has to be Internet facing
  • Your Most Recent Version of Exchange Server has to handle Client Connections
  • Your Most Recent Version of Exchange Server has to be responsable for Exchange Web Service and Autodiscover.
  • An Hybrid Server is a Exchange Server, there is not Hybrid Role (Reminder)

When these rules are followed, random issues related to Exchange Web Service are avoided but sometime I see  Hybrid Design like the following trying to workaround these rules:

Please find an example of wrong Design (when not following theses recommandations)hybrid-exchange-2010_v2-1

The reason of implement such design are often explained by :

  • Downtime when you need to update\replace  Exchange Server by New Hybrid Server
  • Customer would like to Test Hybrid Configuration without changing the production environment

This kind of design will never provide the customer will the full experience of Hybrid configuration, but with ramdom Exchange Web Service issues.

So, why this design is wrong, first and easy answer, this design does not follow the basic recommandation we provided earlier. why this recommandations is so important. First these recommandations garantee that your internet facing \Client Access Server can handle any client connection and\or Request. not following this recommandation will create the following issues:

  • If the Hybrid Server is not Internet facing for Client Access Server Request:
    • Autodiscover Service will pointing on legacy  Exchange Server endpoint
    • Autodiscover will provide Exchange Web Service Endpoint pointing to legacy Exchange Server endpoint
  • Legacy Exchange Server won’t be able to provide accurate information.
    • Regarding Exchange Online

So to understand technically what happens, we will have to respond two questions:

  • How Outlook will retreive the required information to connect to Exchange Server?
  • How Exchange Web Service informations are retreived?


After reading serveral articles about how Autodiscover is working with Exchange 2007/2010/2013/2016 &  some articles how outlook will use Autodiscover Service, please  find some of these articles.

  • How Free busy information is retreive:
    • Exchange 2010 (Availability Service Process Flow): link
    • Methods used to Retreive Free\busy information :Link
    • How Free\Busy is retreive: Link
  • How autodiscover is working for Exchange Server(On-premise):
    •  Exchange 2007:Link
    • Exchange 2010:Link
    • Exchange 2013: Link
    • Exchange 2016: Link
  • How Autodiscover is working for Office 365 mailbox:
    • autodiscover using TargetAddress:Link
    • Office 365 autodiscover lookup Process: Link
    • Exchange 2013 Cross-forest moves: Link
  • How  Autodiscover is working in Cross-forest or multi-site environment
    • Exchange 2016 Coexistence:Link

The responses of theses two questions after reading these articles is Autodiscover.

Since Exchange 2007 SP1, Autodiscover is becoming the most important service for outlook client in Exchange environment. This service is used by outlook 2007 and above to:

  • Outlook profile creation
  • Outlook will use autodiscover to retreive Endpoint connections for following services:
    • Where the mailbox is location (Mailbox Servers)
    • Free\Busy Information
    • Out Of Office
    • OAB
    • Shared Mailbox\Archive Mailbox,…

with Exchange 2013 SP1, Autodiscover is now the only recommended way to configure & update outlook configuration.

Now, We know in an office 365 Hybrid environment Autodiscover design is as important as the Hybrid Server Design.

How to Design Autodiscover in Hybrid Environment, I have try to find some official Article. my findings are only two articles

  • Microsoft Exchange Server Deployment Assistant: Link
  • Create DNS Record for Office 365(not for hybrid): Link

These articles do not provide enough information or guidance for a design.

We will try to provide all information you need to make this design.

In order to provide a proper design, we need to know how autodiscover is working in an Hybrid environment:

  • Where Autodiscover  information are published:
    • Publish information using Service Connection Point (SCP) in AD
      • A SCP is created when a  Client Access Server  is installed
      • ServicebindingInformation of SCP will be replicated between GC
    • Outlook Client will then try hard coded specific URL if failed to retreive information from SCP
  • How Outlook will try to retreive Autodiscover information:
    1. Internally or domain joined computer:
      1. Outlook will first try to retreive autodiscover information using SCP
      2. Then outlook will try the following url for SMTP domain
    2. Externally or none joined computer:
      1. for external request Outlook will make DNS request to try to resolve the following url:
        3. SRV Record – allows to redirect your request to url
  • How outlook will Retreive autodiscover information in Office 365 Hybrid environment:
    1. Autodiscover endpoint located using SCP query
    2. Client attempts first URL
    3. Authentication successful to Autodiscover web service using logged on credentials
    4. Autodiscover unable to answer query as mailbox in O365. 0x800C8205,  Redirect to TargetAddress
    5. Start from beginning using new SMTP address:
    6. Autodiscover starts over. Endpoint located using SCP query
    7. Client attempts first URL
    8. Authentication successful to Autodiscover web service using logged on credentials
    9. Autodiscover unable to answer query as mailbox in O365.  Status code:  0x800C8205
    10. DNS lookup process starts: root domain lookup
    11. Root domain lookup fails with status: 12007
    12. Root domain lookup fails with status: 0x8004005
    13. DNS lookup process continues to next namespace:
    14. Lookup fails with status: 12029
    15. Lookup fails with status: 0x800C8203
    16. Local XML file lookup
    17. Local XML file lookup fails with status: 0x8004010F
    18. HTTP Redirect check to:
    19. Client redirected to shared autodiscover namespace:
    20. Client submits Autodiscover request
    21. Authentication needed – 401 response – you request to provide an authentication
    22. HTTP 302 redirect
    23. Autodiscover failed with status: 0x800C8204 – redirect was received to a more accurate location
    24. Autodiscover sent to accurate location
    25. Autodiscover request issued to
    26. Authentication needed – 401 response
    27. Authentication successfully provided
    28. Autodiscover request submitted
    29. Autodiscover XML data successfully received

This information is from Rhoderick Milne [MSFT] Article

regarding to all this informations and every  articles I have mentioned in this article, I will try to provide you a comprehensive list requirements, best pratices and recommandation for Autodiscover in Hybrid Environment:

  • Active Directory:
    • SCP should point to  Hybrid Server
      • A certificate should have this entry
    • On-premise User should with Exchange Online mailbox should have:
      • a remote mailbox
      • Primary SMTP Address
      • RemoteRoutingAddress (TargetAddress Attribute)
  • Autodiscover  URL should be set to point on Hybrid Server:
    • Public DNS:
      • A Record  pointing to External (public) Ip Address of Hybrid Server
      • Cname for autodiscover point to A Record of you Hybrid Server
      • in a Hybrid Scenario, the autodiscover has to point on your On-premise environment, to be more precise has to point to your Hybrid Server.
    • Internal DNS:
      • A record point to IP address of Hybrid Server
      • This Entry has to match the url provide in SCP
    • Certificate should have SCP entry, Autodiscover url publish by Hybrid Server.
  • Hybrid Server has to be:
    • Internet facing for Autodiscover
    • Internet facing for Client Access role
    • has to run the last version of Exchange in the organization

In a Hybrid Environment, sharing policies or Federation are also being used for Free\busy sharing between Office 365 users & on-premise User. this subject will be discussed in the futur article.



How to link you another on-premise user to Azure AD object using immutableID(Hard Match)


Now, with office 365 in Hybrid Mode we need to be able to addresse old scenario like:

User1 left the company and User2 need to be reconnected to user1 Office 365 User (onedrive,Mailbox,sharepoint,…)

User1 has been deleted and you want to reconnect a new User(User2) to the Office 365 User

User1 has been moved from domain1 to Domain2, for administrative reason a new User (User2) has been created in Domain2.

In order to address these scenarios but also the need to disconnect an office 365 user to on-premise user and reconnect another on-premise user in a Hybrid Environment,  we can use the following procedure called Hard Match:

Some additionnal information, in this list of scenario we are making the following assumptions:

  1. User1 and User2 are in the Same AD Forest, this is important because that s means the User will have two diferent ObjectGUID
  2. AD Connect version is matter now because the Hard Match may not work with every version of AD Connect.  Please check  AD Connect Fixes & Improvements
  3. AD Connect is used as Directory synchronization between On-premise AD and Azure AD (Ofice 365)
  4. AD Connect is using the ObjectGUID as SourceAnchor (ImmutableID)
  5.  the environment is running Exchange 2013 RU11

Now, please find the step by step to disconnect and reconnect an user to Office 365 using Hard Match:

  1.  Information regarding User and environment:
    • User1 will be Source\User1
      • User1 UPN:
      • Source User1 has a Remote Mailbox (Office 365 Mailbox)
    • User2 will be Destination\User2
      • is not sync to Cloud
  2. Update Source\User1 Attributes:
    • Mail
    • TargetAddress
    • UPN: in this Procedure User2 will use the User1 UPN because PrimaryEmail Address has to match will the UPN. FYI we can only have one as UPN in this forest. so the UPN will have to be changed. So we will change to so this UPN can be assign to Destination\User2.
    • ProxyAddresses: Save the ProxyAddresses Attribute then clear.
  3. Disable Source\User1 Remote Mailbox
  4. remove Source\User1 from AD Connect Sync Scope
    • Update Attribute use by AD Connect to Sync or No Sync a user to office 365
    • Wait for a Delta sync Cycle to be completed
  5. Disable Source User
  6. Restore MSOL User (Source\ )
    • Restore MSOL User(
    • Set Restored MSOL User ( with immutableID to Null
    • Calculate Destination User (Destination\User2) ImmutableID
    • Set ImmutableID of MSOL User ( to ImmutableID of Destination User (Destination\User2)
    • Change MSOL User UPN to
    • Delete MSOL User (
  7. Update Destination\User2 Attributes
    • Update Destination User2 to match Source\User1 Attributes
    • Update Destination\User2 UPN to ( Please be sure to wait for AD Replication to occur, or it will failed if it found a user with same UPN in the forest)
    • Enable Destination User
  8. Add Destination\User2 to AD Connect Sync Scope
    • Update Attribute use by AD Connect to Sync or No Sync a user to office 365
  9. Enable Remote mailbox for Destination\User2
    • Enable Remote Mailbox for Destination\User2
    • Reattach the Archive mailbox to Office 365 mailbox (if previously attach to Source\User1)
    • Assign the ProxyAddress previously set for Source\User1
    • Update TargetAddress
  10. When complete wait for AD Connect Sync Cycle ( I will recommand to run at least 2 cycle, some time the hard Match is not working at the first Sync)

Important Information, while using this procedure you need to be aware between step 3 and Step 10, the mailbox is disabled so NDR will be receive if you try to send a mail to Office 365 mailbox. so the downtine will be :

  1.  1 AD Connect Sync Cycle for Step 3 to wait for:
    • Wait for Delta Sync, Default setting: 30 min
    • Wait for Delta Sync Cycle to completed (running time): Less than 30 min
  2. 2 AD Connect Sync Cycle for Step 10 to wait for

So with the default configuration of AD Connect, the Maximum downtine if you are not able to start a Delta Sync will be  3 hours.


This Article will be updated with Powershell CMDLets.


Relation between ObjectGUID, SourceAnchor & ImmutableID in Office 365 Hybrid Environment


I have provided you  with a script how to convert the ObjectGUID of AD User to ImmutableID, but I have forgot to explain you the relation between them.

In Hybrid environment with  AD Connect using to sync On-Premise user to Azure AD, with AD Connect set with the default setting:


  1. AD Connect will calculate the source Anchor based on ObjectGUID
  2. AD Connect will also sync the User principal Name using the UserPrincipalName Attribute of the user

Please find the step AD Connect will use to Convert ObjectGUID to ImmutableID:

  1. AD Connect will retreive the User ObjectGUID Attribute through AD Connector
  2. AD Connect will then convert the ObjectGUID to CloudSourceAnchor(in the Metaverse)
  3. AD Connect will then export CloudSourceAnchor(Metaverse) to SourceAnchor(AAD Connector)
  4. The AAD Connector will export SourceAnchor to Azure AD
  5. The SourceAnchor will be saved as ImmutableID Attribute of Azure AD Object.


I hope this information will help to better understand how your AD Connect is working.