Working with a client to move users between a Lync 2010 pool and a new Lync 2013 pool, a collegue of mine came across the following error:

Unable to connect to some of the servers in pool “poolname” due to a Distributed Component Object Model (DCOM) error. Verify that Front End service is running on servers in this pool. If the pool is set up for load balancing, verify that load balancer is configured correctly.

Trying the same move from the Lync Management Shell had the following error:

Cannot find Registrar pool. Verify that “poolname” is a valid registrar pool.

Searching the net for the error yielded fixes if the pool was behind a hardware load balancer or messing with the dcomcnfg tool.

• http://blogs.technet.com/b/dodeitte/archive/2010/12/19/issue-when-moving-legacy-users-to-a-lync-server-2010-pool-using-hlb.aspx
• http://www.bibble-it.com/2011/03/22/unable-to-move-lync-user-dcom-error

Unfortunately neither solution work in our scenario. To resolve the error, the Local Security Policy of the servers reported in the error needed to be modified.

Fix:

1. Click “Start” -> “Run” -> Type “secpol.msc” -> Click “OK”
2. Expand “Local Policies” -> Click on “Secuirty Options”
3. Double Click “DCOM: Machine Launch Restrictions in Security Descriptor Definition Language (SDDL) syntax” Policy
4. On the “Template Secuirty Policy Setting” Click “Edit Security”
5. Click “Add” -> Type “RTCUniversalServerAdmins” -> Click “OK”
6. Check all the Allow boxes for the RTCUniversalServerAdmins account -> Click “OK”
7. Log off and on

All credit goes to Korbyn for finding the fix.

During a recent project deploying SBAs for Lync 2013, I came across an issue where the Lync Front-End service would not start.

The SBA was an AudioCodes Mediant 1000 SBA running version 1.1.12.0

Going through the Deployment Web GUI, the error would occur at “Start MCS Services”

Looking at the Deployment Log, the Front-End (RTCSRV) service would timeout.

Connecting to the SBA using RDP and reviewing Event logs showed:

• System Event Log with Event ID: 7024

• Lync Server Event Log with Event ID: 50006

• Application Event Log with Event ID: 1000

• Windows Fabric Admin Event Log with Event ID: 2564

The root issue was the Windows Fabric service would not stay running and seemed to be corrupt in the SBA image.

To Repair the Window Fabric Component:

1. Stop Lync Services
   
2. Remove Windows Fabric application
   

3. Don’t Restart
4. Install Windows Fabric *** C:SourcesWindowsFabric
   

5. Restart Windows
6. Confirm Windows Fabric Service starts
   
7. Confirm all the Lync Services start *** Lync Server Front_end will take a while to go from Starting to Start
   

Looking at TechNet, the same issue was reported with an OfficeMaster SBA from Ferrari Electronic: http://social.technet.microsoft.com/Forums/en-US/81699c3a-969b-4e2a-acc5-2148537928d3/windows-fabric-host-service-and-rtcsvc-not-starting-on-lync-2013-sba?forum=lyncdeploy

While deploying a Lync 2013 pool for a client, we ran into issues publishing the new Lync Topology and installing databases.

Looking at the Create database log, the issue occurs while the database owner is being set:

Setting owner for database xds to sa.
Exception Stack:
Type: Microsoft.SqlServer.Management.Smo.FailedOperationException, Message: Set owner failed for Database ‘xds’.
Type: Microsoft.SqlServer.Management.Smo.SmoException, Message: An exception occurred in SMO.
Type: System.ArgumentException, Message: The login ‘sa’ does not exist on this server.

Using SQL Management Studio we see the problem database stuck in “Restricted User” mode and find that the SQL team’s best practice is to rename the SA account.

The quick fix is to simply rename the account back to SA, drop the database and publish the Topology again.

Keep in mind if you don’t drop the blank problem database, you will see another error while publishing the Topology:

An internal error has occured while trying to create or update the database.
Error: Database xds could not be updated because it requires major upgrade from version 0 to version 10.

After recreating the issue in my lab

• I have found that no matter which new database(Central Management/Enterprise Edition Back End/Monitoring/Archiving/Persistent Chat) is deployed the SA account must exist and can’t be renamed.
  
• The SA account doesn’t have an impact on deploying Lync Monitoring Reports.
• Although during normal operation renaming the SA account back to something else doesn’t impact Lync services, going through the database update procedure that is part of installing Lync Server Cumulative Updates ( http://support.microsoft.com/kb/2809243 ) will run into the same problem.
  

While verifying a new conference bridge for a client, I came across a phantom conference bridge entry listed on the Lync Server Dial-in page but not on the Outlook meeting invites.

Using Get-CsDialInConferencingAccessnumber cmdlet which lists information about all dial-in conferencing numbers configured in an organization and the region parameter to search for the specific entry. A warning was returned saying the contact object for the entry could not be found. Looks like the bridge was leftover from the OCS days. Record the “Line URI” listed in the warning message.

get-csdialinconferencingaccessnumber -region "<RegionName>" -Verbose |FL

First step was to recreate the region that was associated to the phantom bridge. This was done by creating a new User Dial Plan with the “Dial-in conferencing region” exactly matching what was listed on the Dial-in page for the phantom bridge.

Next create a new Dial-in Access Number where the “Line URI” matched what was returned in the warning from the get-csdialinconferencingaccessnumber, and add the “Associated Region” that was created in the last step.

Running the same get-csdialinconferencingaccessnumber as before will now display the contact object.

Now delete the new Dial-In Access Number and the User Dial Plan that was created for the phantom bridge and invoke CMS replication (invoke-csmanagementstorereplication). Running the get-csdialinconferencingaccessnumber command will now show no object and the phantom bridge will be removed from the Dial-in page.

As a side note you may have noticed in the screenshots I’m using csdialinconferencingaccessnumber vs. get-csdialinconferencingaccessnumber, not sure if the cmdlet was just too long to type or it was late. After taking to Twitter about this “Feature”, Ravikanth Chaganti recommended I trace what’s actually happening.

trace-command -ex { csdialinconferencingaccessnumber } CommandDiscovery -PSHost -Verbose

One of the last steps when PowerShell is unable to find a command is to prepend the command with “get-“.

Update: Looks like Andrew Morpeth blogged about a similar stuck conferencing number, see his post: https://ucgeek.co/2015/03/remove-stuck-conferencing-number-lync/

While testing a new conference bridge that was added to an existing Lync Server 2013 deployment, the Lync client would report the bridge number “is in another call”.

Sifting through server side logs the call was bouncing around like a hot potato between the Front End and Edge servers. The Front End would try to look up the object and return a “SIP 404 Not Found” (ms-diagnostics: 1003;reason=”User does not exist”;destination=”RtcApplication-86d18fa5-3a18-43f6-aef8-c7b3c76c086b@domain.com) and pass the request on to the Edge. The Edge would pass back the same SIP 404 error.

On to some trouble shooting steps… Removing and recreating the bridge, restarting the Conferencing Attendant and Conferencing Announcing Services and finally a Front End pool restart. No luck, back to the server logs. This time instead of searching a specific call, I broadened the filter to “RtcApplication”. Finding that all Application Contacts (RGS Presence Watcher, Call Park Service, CAAPrivateContactObject, Conferencing Announcement Service …) in the environment were giving 404 Not Found/User does not exist errors.

Digging into Active Directory(AD) (I like to use Active Directory Explorer from Sysinternals: https://technet.microsoft.com/en-us/sysinternals/bb963907), I could see that all the corresponding Application Contacts exist from the server log. The msRTCSIP-ApplicationContacts container class was originally added in Office Communications Server 2007 R2 and a good TechNet article about the contact objects can be found at https://technet.microsoft.com/en-us/library/dd572377(v=office.13).aspx. Haven’t found an updated version for Lync or Skype for Business.

Now is probably a good time to inform you that in this environment the Lync Servers were installed into a Child domain with the User objects and the Root domain was empty. I wouldn’t say it was a Null root as the Application Contacts were in the root domain. So after spinning cycles verifying Schema, Forest and domain(s) prep, permissions and passing AD queries between the domains every thing seemed correct with no errors.

Next to check out the User Replicator, as this periodically retrieves current user account info from Active Directory and synchronizes the information with the Lync Server’s user data database.

Get-CsUserReplicatorConfiguration

At some point the client scoped Lync to be restricted to the Child domain, not realizing that the Application Contacts were outside this scope. This wasn’t a problem in an Instant Message and Presence world, but sure was after adding conferencing and enterprise voice! Luckily three commands and the conference bridge was functional.

Set-CsUserReplicatorConfiguration -Identity global -ADDomainNamingContextList $Null
Update-csuserdatabase
Update-csaddressbook

While working on a global Skype for Business deployment, I was moving a test user account between 6 pools and ran into DCOM errors when trying to move the account to two of the pools.
The specific error shown in the Skype for Business 2015 Control Panel:

Unable to connect to some of the servers in pool “poolname” due to a Distributed Component Object Model (DCOM) error. Verify that Front End service is running on servers in this pool. If the pool is set up for load balancing, verify that load balancer is configured correctly.

Wait a minute, haven’t I already blogged about this: http://realtimeuc.com/2013/10/dcom-move-lync-user/ well that Lync 2010 Server fix didn’t work. I noticed although the error was the same in the Control Panel, I had no issues using the Management Shell to move to and from the problem pools.

Time to look at the even logs… On the requesting Front End I would see a System Event Log ID: 10006 for each Front End in the destination pool.

On each Front End in the destination pool I would see a System Event Log ID: 10016.

Now to fire up dcomcnfg and expand: Component Services, Computers, My Computer, DCOM Config.
The Application ID in the event log {DAA77B0D-C664-437C-8B37-22CDC053B961} is the RTC Store Access Interface Class.

Right click RTC Store Access Interface Class and select Properties, Security Tab and Edit the Launch and Activation Permissions.

Problem Front End:

Successful Front End:

So I added the local server’s RTC Local Administrators, RTC Local User Administrators and RTC Server Local Group with Allow Local and Remote Activation on each Front End in the problem pools.

Didn’t need a reboot or service restart as moving the user from the Control Panel was instantly successful.

I tried to determine the root cause of the permission discrepancies for the one DCOM object. All 15 Skype for Business Front Ends were deployed identically and in the same Active Directory Domain and Organizational Unit, still unsure why 6 had the issue.

While working with a Third-Party hosted Skype for Business environment leveraging Exchange Online for Email/Voicemail, I came across an issue where federation worked except with sip domains hosted in Office 365.

Looking at the client logs from both sides of the communication in snooper, you can see constant 504 Server time-out entries:

Hosted client to Skype for Business Online trace:

Skype for Business Online to Hosted client trace:

Hosted client to Skype for Business Online 504 error detail:

• 03/18/2016|10:11:04.993 1F2C8:20684 INFO :: Data Received -209.197.182.66:443 (To Local Address: 172.17.1.57:52011) 832 bytes:
  03/18/2016|10:11:04.993 1F2C8:20684 INFO ::
  SIP/2.0 504 Server time-out
  ms-user-logon-data: RemoteUser
  Authentication-Info: TLS-DSK qop=”auth”, opaque=”8AE5D494″, srand=”194B5A1A”, snum=”41″, rspauth=”e035cf3791629bb547cb087dde5f41d0f6ccd1d7″, targetname=”hostedfe.hdomain.com”, realm=”SIP Communications Service”, version=4
  Via: SIP/2.0/TLS 172.17.1.57:52011;received=72.29.243.33;ms-received-port=24342;ms-received-cid=1200300
  Content-Length: 0
  From: “Michael LaMontagne”michael@hdomain.com>;tag=d141844893;epid=b5f478829d
  To: michael@lamontagnesolutions.com>;tag=9B47C2FDF604DCF0E6773B737ABC04C6
  Call-ID: 07090444edc64a189aa6ce0a2c4700c4
  CSeq: 1 INVITE
  ms-diagnostics: 1063;reason=”Cannot route to blocked IM Service Provider”;domain=”lamontagnesolutions.com”;fqdn1=”sipfed.online.lync.com:5061″;source=”sip.hdomain.com”
  Server: RTC/6.0
  03/18/2016|10:11:04.993 1F2C8:20684 INFO :: End of Data Received -209.197.182.66:443 (To Local Address: 172.17.1.57:52011) 832 bytes

Skype for Business Online to Hosted client 504 error detail:

• 03/18/2016|10:05:22.667 2148:32C4 INFO :: Data Received -132.245.1.32:443 (To Local Address: 10.100.30.182:57915) 1114 bytes:
  03/18/2016|10:05:22.667 2148:32C4 INFO ::
  SIP/2.0 504 Server time-out
  ms-user-logon-data: RemoteUser
  Authentication-Info: TLS-DSK qop=”auth”, opaque=”A8BC1734″, srand=”6BA9FCBC”, snum=”45″, rspauth=”3e7612007482289c34a09d47a3090a75be97e792″, targetname=”BL20A09FES13.infra.lync.com”, realm=”SIP Communications Service”, version=4
  Via: SIP/2.0/TLS 10.100.30.182:57915;received=10.8.148.8;ms-received-port=57915;ms-received-cid=4EBE400
  Content-Length: 0
  From: “Michael LaMontagne”michael@lamontagnesolutions.com>;tag=d97c3dc345;epid=177375e93d
  To: michael@hdomain.com>;tag=B3656182BBE3D680E2A42799A4ACBD34
  Call-ID: 8cd5cb094b5f4c3e937c2553e35ba11f
  CSeq: 1 INVITE
  ms-diagnostics: 1047;reason=”Failed to complete TLS negotiation with a federated peer server”;fqdn=”sip.hdomain.com:5061″;ip-address=”XXX.XXX.XXX.XXX”;peer-type=”FederatedPartner”;winsock-code=”10054″;winsock-info=”The peer forced closure of the connection”;source=”sipfed1A.online.lync.com”
  Server: RTC/6.0
  ms-edge-proxy-message-trust: ms-source-type=AutoFederation;ms-ep-fqdn=sipedgeBL20A.infra.lync.com;ms-source-network=federation;ms-source-verified-user=verified
  03/18/2016|10:05:22.667 2148:32C4 INFO :: End of Data Received -132.245.1.32:443 (To Local Address: 10.100.30.182:57915) 1114 bytes

Much time was spent digging into Federation settings, DNS, certificates and firewalls for both parties. It dawned on me that Skype for Business Online is ignoring Federation SRV records when communicating with verified Custom domains in Office 365. Which is fine if you have Hybrid configured for either your On-Premises or hosted Skype for Business environments, which was not this case.

This fix was to remove the Skype for Business domain purpose for the Third-Party hosted domain:

Open Office 365 Admin center, under Domains click Domain settings:

Click on Change domain purpose:

Uncheck Skype for Business for instant messaging and online meetings, complete the steps in the wizard:

I’ve come across an interesting bug in the Admin Center Preview portal in-regards to the domain purpose changing without reason. This is further to my recent blog posting: http://realtimeuc.com/2016/06/federation-fails-with-o365/, which required disabling the Skype for Business domain purpose for domains using On-Premises or Third-Party hosted environments without hybrid configured to resolve federation issues.

First you’ll find there isn’t a way in the Admin Center Preview portal to view or modify the domain purpose for Custom domains.

You can use the Azure AD PowerShell to check domain purpose (https://msdn.microsoft.com/en-us/library/jj151815.aspx):

As we can see my custom domain is only enabled for Exchange Online and Intune. Note: At this time you can’t change the domain purpose/capabilities using PowerShell.

Now to the bug…. While using the Admin Center Preview portal, go under Settings, Domains, select your Custom domain, click DNS Management and use any option (Next/Back/Exit), you now have enabled all the domain purpose items without notification or warning.

At this point the only way to modify the domain purpose is to revert back to the old Admin portal and follow the steps outlined in http://realtimeuc.com/2016/06/federation-fails-with-o365/:

Status:

Reported to Office 365 support.

Hopefully this blog post will help any others out there having an issue understanding what language is being used in the voice prompts and email transcriptions for Microsoft’s Office 365 Cloud PBX Voicemail (Azure Voicemail).

Until last week the language used in Voicemail services for Office 365 Cloud PBX had to be configured at the organization level within the company profile. This language setting was for all users in the Office 365 tenant and couldn’t be adjusted per user: https://support.office.com/en-us/article/Change-the-default-language-for-voicemail-greetings-and-emails-in-your-organization-820c3892-1b7e-47d3-ae8d-6e27e7cbcf38

So what happened last week? Well my clients piloting Cloud PBX started reporting users were hearing Portuguese/Spanish/Catalan when calling into their Voicemail or external callers trying to leave a message. After opening a support ticket with Microsoft and looking at every possible language setting on the user/mailbox/tenant which was all English, we found that a new undocumented feature was released… The feature was the ability to change the default language on the individual user level. I’m sure at the time of posting this blog the feature will appear on the Office 365 Road Map: https://fasttrack.microsoft.com/roadmap

This new feature is triggered based on the “PreferredLanguage” user attribute, which must be updated in Active Directory if doing directory synchronization. If “PreferredLanguage” is empty the “UsageLocation” user attribute is used instead.

I’m sure you’re thinking what does this have to do with Catalan? Well us Canadians, we are a proud bunch and always select “Canada” when able. An example of this is when licensing a user in Office 365.

If you’ve played with licensing in PowerShell for Office 365 you would have used:

 set-MsolUser -UserPrincipalName <UserUPN> -UsageLocation CA

But wait “CA” is a the country identifier for Canada, but a Language identifier for Catalan (Spain)!

So we looked at the following ways we could quickly update all our client’s users.

From Office 365 PowerShell:

get-MsolUser | set-MsolUser -UsageLocation US

or

Active Directory PowerShell

Get-AdUser -Filter * -ResultSetSize 9999 | Set-ADUser -Replace @{‘preferredLanguage’=”EN-CA”}

Of course the client opted for the second option, keeping everything labeled Canada. Great news is after a directory synchronization, the correct language was used within minutes.

**Update** Microsoft has implemented a fix for this issue and now the voicemail language detection will not leverage the “Usagelocation” if “PreferredLanguage” is empty. If “PreferredLanguage” is empty the Tenant’s “Preferred Language” under the company profile will be used and if that’s not set “English” will be the default.

For me 2016 will be the year Juniper (specifically SRX) attacked Skype for Business desktop sharing. I figured it was time to document at least one of my battles.

Let me first start this post with the fact that I am not a networking expert, but have become an expert at finding network problems and convincing a network resource what they need to do to fix it.

Scenario:
Client reports that Desktop Sharing within a conference works internally but not externally, A/V no issues.

First thoughts:
Since A/V is most likely using UDP and Desktop Sharing is TCP (yes Video Based Screen Sharing (VBSS) prefers UDP, but RDP only uses TCP and RDP signalling is always maintained for failback).

Also we need to focus our efforts around the Edge, as no issues internally.

Next Steps:
History tells me that either the issue is the NAT is not bidirectional, DNS resolution issues on the Edge or Network. Of course the first two are easy to rule out with access to the Edge.

Time to fire my canned email to the client’s network team:
“Skype for Business A/V is not having an issue which is leveraging UDP, we need to look at what could be messing with the TCP stream that Desktop Sharing uses. This almost is 100% a router or firewall configuration and usually due to deep inspection features especially on SSL.”

Well no luck with that email, time to trace and prove.

Client log (Lync-UccApi-0.UccApilog) in Snopper:

CLS logging in Snooper:

Important info:
BYE message indicating that “Call failed to establish due to media connectivity failure when one endpoint is internal and the other is remote”, MediaType of “Application-sharing”. Let’s note the LocalAddress port of 25221 to easily search in the WireShark captures, also the ICEWarn of 0x8000100.

Time to fire up the ICE Warning Flag Decoder (https://gallery.technet.microsoft.com/office/ICE-Warning-Flag-Decoder-97058ef3)

So we have: “there is no direct TCP connection possible”.

On to Wireshark, filtering with “tcp.port==25221”

Client:

Edge:

Juniper SRX:

Important info:
Well I know that the black lines are “BAD TCP” and the red are “TCP Resets”, which always makes me worry. WireShark Packet colorization: https://www.wireshark.org/docs/wsug_html_chunked/ChCustColorizationSection.html

What’s interesting is the Juniper SRX is reporting “TCP Out-Of-Order”, so we know this device is watching the TCP session and resetting what it doesn’t like.

After a bit of googling I came up with a list of settings for the network resource to disable on the SRX:

• tcp-syn-check (TCP Syncro Check:)
• tcp-seq-check (TCP Sequence Check)

Note: How to selectively disable TCP SYN or Sequence checking: https://kb.juniper.net/InfoCenter/index?page=content&id=KB25094&actp=search

Magic! We’re in business and no Desktop Sharing issues externally anymore.

Another one of my battles the client had the TCP syn/seq already disabled, but also had to add:

• tcp-rst-invalidate-session: https://www.juniper.net/documentation/en_US/junos12.1×46/topics/reference/configuration-statement/security-edit-rst-invalidate-session.html

Some settings that others have reported they needed to disable:

• tcp-syn-bit-check
• alg sip enable

Looking at the Juniper forums there’s an open topic on this issue: https://forums.juniper.net/t5/SRX-Services-Gateway/Skype-Business-desktop-sharing/td-p/291272, I don’t have an account with Juniper, so unable to share my findings.

I get the pleasure of working with a verity of Multi-Forest environments, today’s scenario is a hosted Skype for Business environment within a Resource Forest and leveraging Exchange Online. The design is similar to the documentation on TechNet for Configure a Multi-Forest Environment for Hybrid Skype for Business but without the Hybrid of Exchange or Skype for Business.

Modified Topology Diagram:

A Client reported that when creating a new Skype Meeting in Outlook and modifying the meeting options to allow “People I invite from my company” to bypass the lobby, the company invitees are still directed to the lobby for the meeting.

What’s interesting is if the Organizer selects “Anyone from my organization” the invitees from the company bypass the lobby as expected.

First step was to check the Skype for Business client and Exchange connectivity (MAPI/EWS). All good there…

Now to check the Contact Card, yes the sip address is shown.

But wait, what if Outlook is using the Exchange GAL to populate the approved sip address list for the bypass the lobby meeting option? As you can see from the Topology Diagram we aren’t syncing any attributes back to the Account Forest and as such the sip address is not listed in the Exchange E-mail addresses.

Simple enough fix, let’s manually populate the proxyAddresses with the “SIP:usersipaddress“.

Now we see the sip address entry on the Exchange GAL and the user is able to bypass the lobby when listed as one of the “People I invite from my company”.

To Recap:

• When “Anyone from my organization” is selected, a flag is set to match the sip domains configured on the Skype for Business server side.
• When “People I invite from my company” is selected, the Outlook Skype Meeting Add-In populates an authorized list of invitees based on sip addresses listed in the Exchange GAL and not the Contact Card.

A couple weeks ago I came across a client issue enabling CloudPBX for Skype for Business Online users. The correct Set-CsUser command that would normally enable users, was bleeding red errors in PowerShell. The command is pretty straight forward and was successful the day before.
An example command:

set-csuser -identity usersipaddress@domain.com -EnterpriseVoiceEnabled $True -HostedVoiceMail $True -OnPremLineURI tel:+12345678901

Instead of enabling the specific user for CloudPBX, PowerShell returned an “Object reference not set to an instance of an object” error. Stripping down the cmdlet, it was determined that the error occurs when attempting to set the EnterpriseVoiceEnabled parameter on Set-CsUser to $True.

After hitting up the MVP Community, I have found this is occurring globally and randomly within Office365.

Although I can’t explain what triggers the issue, I have come up with the following workaround that can help others that are experiencing the (hopefully temporary) issue that could be preventing CloudPBX deployments.

The fix is to set the HostedVoiceMail parameter on Set-CSUser to $True alone without any other parameters, after doing so wait ~10 seconds and run your original Set-CsUser command setting the Enterprise Voice parameters and all is successful. If HostedVoiceMail is already $True and seeing the issue with EnterpriseVoiceEnabled, toggle HostedVoiceMail to $False and back to $true and should be good to go.

Today I’m shedding light on the SIP 504 response status code while working with Skype for Business Online’s CloudPBX and Cloud Connector Edition (CCE). The “504 Server time-out” error is masked in the Skype for Business Client as “Operation was unsuccessful” during an outbound PSTN call.

Understanding this error can actually become a handy tool in an Administrator’s toolkit (more on this later!).

The key is to view the ms-diagnostics details specific to the SIP response.

• This can be done by opening the .UccApilog file from the client’s Tracing folder (https://support.office.com/en-us/article/Turn-on-error-logs-in-Skype-for-Business-4892fe33-ef40-48e2-9070-5308d2a57c7e) with a text editor or Snooper. Snooper is part of the Skype for Business Server 2015 Debugging Tools (https://www.microsoft.com/en-us/download/details.aspx?id=47263).
• Alternatively, an Administrator could pull the same info using the Get-CsUserSession Cmdlet (https://technet.microsoft.com/en-us/library/mt715516.aspx ) or even better by using my PowerShell GUI for the same Cmdlet (http://realtimeuc.com/2016/10/get-csusersessiongui/).

Let’s look at some scenarios that cause the 504 Server time-out response:

• CCE Edge server outage:
  This will give you an ms-diagnostics message: “Failed to connect to a federated peer server”;ip-address=”X.X.X.X“;peer-type=”FederatedPartner”;winsock-code=”10061″;winsock-info=”The peer actively refused the connection attempt”;source=”sipfedXXX.online.lync.com”.

  The IP address will be the Public IP of the CCE Edge server and the source will be the last hop before the issue, this being SIPFed within Skype for Business Online.

• CCE Mediation server outage:
  This will give you an ms-diagnostics message: “Failed to connect to a peer server”;ip-address=”X.X.X.X“;peer-type=”InternalServer”;winsock-code=”10061″;winsock-info=”The peer actively refused the connection attempt”;source=”cce.domain.com“.

  The IP address will be the Private IP of the CCE Mediation server and the source will be the last hop before the issue, this being the CCE Edge server.

• Attempting to use an onmicrosoft.com domain or a SIP domain not configured within CCE:
  This will give you an ms-diagnostics message: =”Cannot route From and To domains in this combination”;cause=”Possible server configuration issue”;summary=”The domain of the message that corresponds to remote peer (external) is not shared between local and remote deployments”;external-domain=”domain.com“;external-type=”domain-type-local”;internal-domain=”domain.com“;internal-type=”domain-type-local”;source=”sipfedXXX.online.lync.com”.

  The domain referenced will be the CloudPBX user’s SIP domain that is not configured within CCE (onmicrosoft.com domains will never work) and the source will be the last hop before the issue, this being SIPFed within Skype for Business Online.

• Sample of viewing ms-diagnostics message with Get-CSUserSessionGUI.
  

You might be asking yourself how could this 504 response be useful? Unfortunately, I figured this out a little too late a few weeks back.

I had opened a Microsoft Office 365 Support ticket due to an outbound calling issue. Even though I submitted the Client logs, made note the call never reached my CCE and selected CCE when creating the ticket, I was greeted with the canned response:

“We would be requiring the below mentioned logs to analyse the issue causing the problem.

1. SIP Stack
2. S4
3. Mediation
4. Inbound Routing
5. Outbound Routing

Along with these logs we require the Client logs (Tracing Folder) as well as Cloud Connector Configuration ini File.”

I now know that if I simply just turn off my CCE and I don’t get a 504 response from the Client calling outbound, the call is not making it to SIPFed and is failing much earlier within Skype for Business Online.

On July 31, 2017 Microsoft announced the Teams Outlook add-in to schedule meetings: https://techcommunity.microsoft.com/t5/Microsoft-Teams-Blog/Now-available-Outlook-add-in-to-schedule-meetings-in-Microsoft/ba-p/71157

Excited to give this a run through its paces, I noticed Outlook was missing the Teams button under New Items and when creating a new Meeting.

Looking under Outlook Add-ins, I found no entries for Teams. Of course it couldn’t be as simple as enabling it…

Next was to follow some recommendations on how others resolved the Teams Add-in issue:

• Adam Ball: Teams Meeting Outlook plug-in not showing
• Dino Caputo: Teams Outlook Add-In Not Appearing

Basically I spent a day of uninstall / reinstall tag with multiple Office 365 accounts enabled for Teams and matching Outlook profiles. The end result was no different and the Add-in was still not listed.

If you right click the Teams system tray icon you can “Get logs”.

Each time Teams is launched the Outlook Add-in is checked and each time I received “Errorcode:2”, “meeting_addin_registration_failed”.

Sat Aug 05 2017 09:47:23 GMT-0600 (Mountain Daylight Time) <15912> -- info -- Silent update trigger set to false 
Sat Aug 05 2017 09:47:23 GMT-0600 (Mountain Daylight Time) <15912> -- info -- Current Outlook Version: 16 
Sat Aug 05 2017 09:47:23 GMT-0600 (Mountain Daylight Time) <15912> -- error -- Error in registering outlook meeting addin. Errorcode:2 
Sat Aug 05 2017 09:47:23 GMT-0600 (Mountain Daylight Time) <15912> -- event -- name: meeting_addin_registration_failed
result: 2

Attempting to be clever, I used Outlook’s COM Add-ins window and manually added Microsoft.Teams.AddinLoader.dll, which I found under %LocalAppData%\Microsoft\TeamsMeetingAddin. This didn’t work and no additional Add-in showed after clicking OK.

Next I installed Teams on a fresh PC and exported all the Registry Keys I could find referencing “TeamsAddin”:

Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}]
@="FastConnect Class"

[HKEY_CLASSES_ROOT\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\InprocServer32]
@="C:\\Users\\admin\\AppData\\Local\\Microsoft\\TeamsMeetingAddin\\1.0.17186.2\\x64\\Microsoft.Teams.AddinLoader.dll"
"ThreadingModel"="Apartment"

[HKEY_CLASSES_ROOT\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\ProgID]
@="TeamsAddin.FastConnect.1"

[HKEY_CLASSES_ROOT\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\Programmable]

[HKEY_CLASSES_ROOT\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\TypeLib]
@="{C0529B10-073A-4754-9BB0-72325D80D122}"

[HKEY_CLASSES_ROOT\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\Version]
@="1.0"

[HKEY_CLASSES_ROOT\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\VersionIndependentProgID]
@="TeamsAddin.FastConnect"

[HKEY_CLASSES_ROOT\TeamsAddin.FastConnect]
@="FastConnect Class"

[HKEY_CLASSES_ROOT\TeamsAddin.FastConnect\CurVer]
@="TeamsAddin.FastConnect.1"

[HKEY_CLASSES_ROOT\TeamsAddin.FastConnect.1]
@="FastConnect Class"

[HKEY_CLASSES_ROOT\TeamsAddin.FastConnect.1\CLSID]
@="{19A6E644-14E6-4A60-B8D7-DD20610A871D}"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}]
@="FastConnect Class"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\InprocServer32]
@="C:\\Users\\admin\\AppData\\Local\\Microsoft\\TeamsMeetingAddin\\1.0.17186.2\\x86\\Microsoft.Teams.AddinLoader.dll"
"ThreadingModel"="Apartment"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\ProgID]
@="TeamsAddin.FastConnect.1"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\Programmable]

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\TypeLib]
@="{C0529B10-073A-4754-9BB0-72325D80D122}"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\Version]
@="1.0"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\VersionIndependentProgID]
@="TeamsAddin.FastConnect"

[HKEY_CURRENT_USER\Software\Classes\CLSID]

[HKEY_CURRENT_USER\Software\Classes\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}]
@="FastConnect Class"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\InprocServer32]
@="C:\\Users\\admin\\AppData\\Local\\Microsoft\\TeamsMeetingAddin\\1.0.17186.2\\x64\\Microsoft.Teams.AddinLoader.dll"
"ThreadingModel"="Apartment"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\ProgID]
@="TeamsAddin.FastConnect.1"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\Programmable]

[HKEY_CURRENT_USER\Software\Classes\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\TypeLib]
@="{C0529B10-073A-4754-9BB0-72325D80D122}"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\Version]
@="1.0"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\VersionIndependentProgID]
@="TeamsAddin.FastConnect"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}]
@="Connect Class"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\InprocServer32]
@="C:\\Users\\admin\\AppData\\Local\\Microsoft\\TeamsMeetingAddin\\1.0.17186.2\\x64\\Microsoft.Teams.AddinLoader.dll"
"ThreadingModel"="Apartment"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\ProgID]
@="TeamsAddin.Connect.1"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\Programmable]

[HKEY_CURRENT_USER\Software\Classes\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\TypeLib]
@="{C0529B10-073A-4754-9BB0-72325D80D122}"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\Version]
@="1.0"

[HKEY_CURRENT_USER\Software\Classes\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\VersionIndependentProgID]
@="TeamsAddin.Connect"

[HKEY_CURRENT_USER\Software\Classes\TeamsAddin.FastConnect]
@="FastConnect Class"

[HKEY_CURRENT_USER\Software\Classes\TeamsAddin.FastConnect\CurVer]
@="TeamsAddin.FastConnect.1"

[HKEY_CURRENT_USER\Software\Classes\TeamsAddin.FastConnect.1]
@="FastConnect Class"

[HKEY_CURRENT_USER\Software\Classes\TeamsAddin.FastConnect.1\CLSID]
@="{19A6E644-14E6-4A60-B8D7-DD20610A871D}"

[HKEY_CURRENT_USER\Software\Classes\TeamsAddin.Connect]
@="Connect Class"

[HKEY_CURRENT_USER\Software\Classes\TeamsAddin.Connect\CurVer]
@="TeamsAddin.Connect.1"

[HKEY_CURRENT_USER\Software\Classes\TeamsAddin.Connect.1]
@="Connect Class"

[HKEY_CURRENT_USER\Software\Classes\TeamsAddin.Connect.1\CLSID]
@="{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID]

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}]
@="FastConnect Class"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\InprocServer32]
@="C:\\Users\\admin\\AppData\\Local\\Microsoft\\TeamsMeetingAddin\\1.0.17186.2\\x86\\Microsoft.Teams.AddinLoader.dll"
"ThreadingModel"="Apartment"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\ProgID]
@="TeamsAddin.FastConnect.1"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\Programmable]

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\TypeLib]
@="{C0529B10-073A-4754-9BB0-72325D80D122}"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\Version]
@="1.0"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{19A6E644-14E6-4A60-B8D7-DD20610A871D}\VersionIndependentProgID]
@="TeamsAddin.FastConnect"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}]
@="Connect Class"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\InprocServer32]
@="C:\\Users\\admin\\AppData\\Local\\Microsoft\\TeamsMeetingAddin\\1.0.17186.2\\x86\\Microsoft.Teams.AddinLoader.dll"
"ThreadingModel"="Apartment"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\ProgID]
@="TeamsAddin.Connect.1"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\Programmable]

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\TypeLib]
@="{C0529B10-073A-4754-9BB0-72325D80D122}"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\Version]
@="1.0"

[HKEY_CURRENT_USER\Software\Classes\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\VersionIndependentProgID]
@="TeamsAddin.Connect"

[HKEY_CURRENT_USER\Software\Microsoft\Office\15.0\Outlook\Resiliency\DoNotDisableAddinList]
"TeamsAddin.Connect"=dword:00000001

[HKEY_CURRENT_USER\Software\Microsoft\Office\16.0\Outlook\Resiliency\DoNotDisableAddinList]
"TeamsAddin.Connect"=dword:00000001

[HKEY_CURRENT_USER\Software\Microsoft\Office\Outlook\Addins\TeamsAddin.Connect]
"Description"="Microsoft Teams Meeting Add-in for Microsoft Office"
"FriendlyName"="Microsoft Teams Meeting Add-in for Microsoft Office"
"LoadBehavior"=dword:00000003

[HKEY_CLASSES_ROOT\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}]
@="Connect Class"

[HKEY_CLASSES_ROOT\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\InprocServer32]
@="C:\\Users\\admin\\AppData\\Local\\Microsoft\\TeamsMeetingAddin\\1.0.17186.2\\x64\\Microsoft.Teams.AddinLoader.dll"
"ThreadingModel"="Apartment"

[HKEY_CLASSES_ROOT\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\ProgID]
@="TeamsAddin.Connect.1"

[HKEY_CLASSES_ROOT\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\Programmable]

[HKEY_CLASSES_ROOT\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\TypeLib]
@="{C0529B10-073A-4754-9BB0-72325D80D122}"

[HKEY_CLASSES_ROOT\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\Version]
@="1.0"

[HKEY_CLASSES_ROOT\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\VersionIndependentProgID]
@="TeamsAddin.Connect"

[HKEY_CLASSES_ROOT\TeamsAddin.Connect]
@="Connect Class"

[HKEY_CLASSES_ROOT\TeamsAddin.Connect\CurVer]
@="TeamsAddin.Connect.1"

[HKEY_CLASSES_ROOT\TeamsAddin.Connect.1]
@="Connect Class"

[HKEY_CLASSES_ROOT\TeamsAddin.Connect.1\CLSID]
@="{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}]
@="Connect Class"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\InprocServer32]
@="C:\\Users\\admin\\AppData\\Local\\Microsoft\\TeamsMeetingAddin\\1.0.17186.2\\x86\\Microsoft.Teams.AddinLoader.dll"
"ThreadingModel"="Apartment"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\ProgID]
@="TeamsAddin.Connect.1"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\Programmable]

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\TypeLib]
@="{C0529B10-073A-4754-9BB0-72325D80D122}"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\Version]
@="1.0"

[HKEY_CLASSES_ROOT\Wow6432Node\CLSID\{CB965DF1-B8EA-49C7-BDAD-5457FDC1BF92}\VersionIndependentProgID]
@="TeamsAddin.Connect"

Download: TeamsAddin Registry keys (Use at your own risk, update the DLL paths, version and always backup your Registry before importing).

After manually importing the Teams Add-in Registry keys, the Teams client no longer reported an error and was showing success.

Sat Aug 05 2017 13:18:44 GMT-0600 (Mountain Daylight Time) <10432> -- info -- Silent update trigger set to false 
Sat Aug 05 2017 13:18:44 GMT-0600 (Mountain Daylight Time) <10432> -- info -- Current Outlook Version: 16 
Sat Aug 05 2017 13:18:44 GMT-0600 (Mountain Daylight Time) <10432> -- info -- Outlook meeting addin registered success 
Sat Aug 05 2017 13:18:44 GMT-0600 (Mountain Daylight Time) <10432> -- info -- Setting logged out state to false 
Sat Aug 05 2017 13:18:44 GMT-0600 (Mountain Daylight Time) <10432> -- info -- App initialized, enabling close to minimize. 
Sat Aug 05 2017 13:18:44 GMT-0600 (Mountain Daylight Time) <10432> -- event -- status: success

After opening Outlook, I still didn’t see the Teams buttons but now the Teams Add-in was listed under Inactive!

Attempting to enable the Teams Add-in from the COM Add-ins resulted in “Not loaded. A runtime error occurred during the loading of the COM Add-in”.

Looking at the System Event Logs, every time Outlook was launched I received two Error events with Event ID: “10016” and Error: “The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID”. This ended up being a permission whack-a-mole for about an hour with no progress.

Out of ideas, I went into Internet Explorer’s Internet Options, Security and clicked “Reset all zones to default level”. Now going into Outlook I was able to enable the Teams Add-in and was greeted with “Load at Startup”.

Success! Teams buttons visible and functioning.

Today I’m shedding light on the SIP 404 response status code while working with Skype for Business Online’s CloudPBX and Cloud Connector Edition (CCE). The “404 Not Found” error is masked in the Skype for Business Client as “We couldn’t reach +1XXXXXXXXXX” during an outbound PSTN call.

Always view the ms-diagnostics details specific to the SIP response.

• This can be done by opening the .UccApilog file from the client’s Tracing folder (https://support.office.com/en-us/article/Turn-on-error-logs-in-Skype-for-Business-4892fe33-ef40-48e2-9070-5308d2a57c7e) with a text editor or Snooper. Snooper is part of the Skype for Business Server 2015 Debugging Tools (https://www.microsoft.com/en-us/download/details.aspx?id=47263).
• Alternatively, an Administrator could pull the same info using the Get-CsUserSession Cmdlet (https://technet.microsoft.com/en-us/library/mt715516.aspx ) or even better by using my PowerShell GUI for the same Cmdlet (http://realtimeuc.com/2016/10/get-csusersessiongui)

Let’s look at some scenarios that cause the 404 Not Found response:

First, don’t be alarmed when you see many 404 Not found in the .UccApiLog. These are expected to occur when the Skype for Business client is attempting presence subscription or communication to contacts on external domains that don’t have federation configured or for objects that simply don’t exist in a domain.

• User doesn’t exist in your domain:
  This will give you an ms-diagnostics message: “User does not exist”;destination=”user@domain.com“;source=”sip.domain.com”
  
• User doesn’t exist in a federated domain:
  This will give you an ms-diagnostics message: “Previous hop shared address space peer did not report diagnostic information”;Domain=”federateddomain.com“;PeerServer=”sip.federateddomain.com“;source=”sip.domain.com”
  
• User from an external domain and that domain is not configured for federation:
  This will give you an ms-diagnostics message: “Unable to resolve DNS SRV record”;domain=”externaldomain.com“;dns-srv-result=”NegativeResult”;dns-source=”WireQuery”;source=”sip.domain.com”
  
• Inbound PSTN called number that is either unassigned or doesn’t exactly match a User’s Line URI(E.164) in Skype for Business Online:
  This will give you an ms-diagnostics message: “Previous hop shared address space peer did not report diagnostic information”;Domain=”domain.com“;PeerServer=”sipfed.online.lync.com“;source=”sipcce.domain.com”
  

The above is all expected, but what if everything is configured correctly and you experience a “404 Not Found” during an outbound PSTN Call when using CCE?

With a successful outbound PSTN Call to CCE, there will be a SIP “101 Trying” followed by a SIP “101 Progress Report”. The Progress Report will highlight communication between Skype for Business Online and the CCE, showing the call is being passed to “OutboundRouting” indicated by the “Trying next hop” message. From this point on, we know the Call is at the CCE and any SIP Errors would be between the CCE and your PSTN connectivity.

With the “404 Not Found” the call flow is quite a bit different. There is the same SIP “101 Trying”, but this is immediately followed by the SIP “404 Not Found”. What we find is Skype for Business Online is doing a Reverse Number Lookup (RNL) for the dialed number within your domain, of course this doesn’t match as we’re trying to call the PSTN and the call is terminated. The call never attempts Outbound Routing and is not sent to the CCE. That last statement is very important, don’t burn support cycles on the wrong rabbit hole. Most CCE issues would be indicated by a SIP “504 Server time-out”: http://realtimeuc.com/2017/05/sfb-cce-504/.

• Digging into the ms-diagnostics message: “User does not exist”;domain=”domain.com“;source=”sipfed.online.lync.com”
  

Possible fixes:

• Confirm the specific Skype for Business Online user has the HybridPSTNSite configured pointing to CCE, using the Get/Set-CsUserPstnSettings: https://technet.microsoft.com/en-us/library/mt670902.aspx. Original CCE documentation just had you set the CCE at the tenant level using: Set-CsTenantHybridConfiguration -PeerDestination. I believe this parameter is going to be deprecated and the better experience is setting the CCE at the user level. Back in November 2016, a couple customers with working users experienced the CSUserPstnSettings was cleared for all their users.
• User is classified as Hybrid, CCE does not support hybrid users. I use the following command to list hybrid users on a tenant:

  Get-CsOnlineUser | ? {$_.interpretedusertype -like 'hybrid*'} | select displayname,userprincipalname,interpretedusertype
  

• Newly licensed for Cloud PBX, it can take up to 24 hours for a user to be enabled fully: https://support.office.com/en-us/article/Assign-Skype-for-Business-licenses-fd41934d-f2eb-4a1b-89d8-32cb37702b33?ui=en-US&rs=en-US&ad=US. I have seen new users have Inbound calling from CCE works within minutes but Outbound calling takes hours to a full day after the Cloud PBX license was applied.
• Internal issues in Skype for Business Online, everything looks correct, but some internal replication or communication problem within Skype for Business Online is preventing the settings from being used. Best course of action is to check the Service Health dashboard for known issues and to create an Office 365 Support ticket.

A journey for your amusement:

December 2012: I submitted my first Lync Server 2013 (RTM) bug. The problem was when an organization enabled for federation with a conferencing policy allowing to invite anonymous users and a meeting organizer had an External Access Policy disabling federated user access. The result would be an external participant could join this meeting if using the web app, but would fail to join if using the desktop client.

April 2015: Scott Stubberfield posted: Anonymous join from Skype for Business and Lync clients. Working directly with Scott, we determined the issue was due to federation being disabled and the external participant was receive a SIP 403 response code, the desktop client was only configured to attempt anonymous join if it received a 504 or 404 response code.

October 2016: I finally got around to blogging about the issue as another client brought up the scenario again: Skype for Business anonymous join failure when meeting organizer is disabled for federation.

June 2017: Full circle, I was working on a Lync 2013 to Skype for Business upgrade project for the same customer that I discovered the bug back in 2012. Even more coincidental, Scott reached out asking if I was able to reproduce this issue and show the Development Team the problem in action.

Sept 2017: ~5 years and we have a fix! The Semi-Annual Channel (Targeted) Office 365 Click-to-Run build 8421.2079, released September 12 has the Skype for Business Non-security update: “Fix an issue where anonymous join users are blocked from joining when federation is disabled and anonymous join is not explicitly blocked by meeting organizer”. A little disappointed it’s not bold or flashing red text, but I’ll take it!

Federated user access disabled:

Allowed to invite anonymous users:

External participant:

• Attempts to join meeting, gets a 403 Forbidden due to Organizer not enabled for federation
• Client automatically sends a new invite with GUID@anonymouse.invalid and joins meeting as a Guest

Let’s just assume that a Skype for Business Cloud Connector Edition (CCE) has been deployed successfully, the steps to configure an online hybrid Mediation Server (Configure online hybrid Mediation Server Settings) was completed and at one time PSTN calls could be escalated to an Audio Conference. Fast forward a few months and users report PSTN call escalation fails every time.

Always view the ms-diagnostics details specific to the SIP response.

• This can be done by opening the .UccApilog file from the client’s Tracing folder (https://support.office.com/en-us/article/Turn-on-error-logs-in-Skype-for-Business-4892fe33-ef40-48e2-9070-5308d2a57c7e) with a text editor or Snooper. Snooper is part of the Skype for Business Server 2015 Debugging Tools (https://www.microsoft.com/en-us/download/details.aspx?id=47263).
• Alternatively, an Administrator could pull the same info using the Get-CsUserSession Cmdlet (https://technet.microsoft.com/en-us/library/mt715516.aspx ) or even better by using my PowerShell GUI for the same Cmdlet (http://realtimeuc.com/2016/10/get-csusersessiongui)

After reviewing the session details the dreaded “404 Not Found” error with the ms-diagnostics details “User Does not exist” appears. This shouldn’t be confused with the same error when just making a normal call via Microsoft Phone System: http://realtimeuc.com/2017/09/sfb-cce-404/.

First, make sure your CCE hasn’t drifted into the future and time is out of sync, Lee Ford has a nice write up on this scenario: https://www.lee-ford.co.uk/unable-to-transfer-or-conference-pstn-calls-with-cloud-connector-edition-cce/

Next, check the health of your Hybrid Mediation Server object. I use the following PowerShell One-Liner to dump a list of all my CCEs and check for any user objects created for use as the Hybrid Mediation Server:

Get-CsHybridPSTNSite | foreach {Get-CsOnlineUser -Filter "HostingProvider -like '*$($_.edgefqdn)*'" | Get-CsHybridMediationServer}

If you get no object, that means someone had deleted the user account.

In my case an Admin rolled out an auto Office 365 licensing script and even though the account had originally met the requirement: “Do not assign any Office 365 licenses (such as E5) to the account you create. Wait for Office 365 AD sync to complete”. Apparently that requirement is for perpetuity and if the account is licensed for Skype for Business Online the AccessProxyExternalFqdn changes from the Edge FQDN of the CCE to “sipfed.online.lync.com” breaking the Hybrid Mediation Server functionality. You do get a nice warning off the Get-CsHybridMediationServer cmdlet (“cannot be set as Hybrid Mediation Server”). Simple fix is to unlicense the account and run the original Set-CsHybridMediationServer.

Unfortunately, I found out that after 30 days the licensing deprovisioning process actually changes the Skype for Business Online user account object to be disabled, again breaking the Hybrid Mediation Server functionality (“not set as valid Hybrid Mediation Server”). Run the original Set-CsHybridMediationServer one more time and you’re back in business. Also a great time to hunt down the Admin that licensed the Hybrid Mediation Server object.

Communications Credits (the add-on formally known as PSTN Consumption), is a subscription add-on to the Microsoft Phone System and/or Audio Conferencing license, enabling the ability to:

• Acquire Toll-Free numbers to use with Audio Conferencing meetings, Auto Attendants, or Call Queues. *Calls are billed per minute and require a positive Communications Credits balance.
• Dial out from an Audio Conferencing meeting to add someone else from anywhere in the world or your mobile phone with the Skype for Business or Microsoft Teams app installed.
• Dial international phone numbers when you have Domestic Calling Plan subscription or Dial out for Domestic/International on a pay per minute once exhausted the minutes included in Calling Plan subscriptions.

More details: https://docs.microsoft.com/en-us/SkypeForBusiness/skype-for-business-and-microsoft-teams-add-on-licensing/what-are-communications-credits

Without Communications Credits you will see an error “Unfortunately, you cannot acquire a toll-free number until you have setup a Skype for Business Consumption billing…” if attempting to add a Toll-Free number under “Add new service numbers” in the Skype for Business Online Admin Center.

Calling Rates for Communications Credits can be found at the bottom of the Calling Plans product details page, fill out the country, currency and download the Excel rate document (SkypeRates.xlsx).

Armed with the above info, you decide that Communications Credits is required for your organization. Under “Purchase services” in the Office 365 Admin Center, if already acquiring licensing from a
Cloud Solution Provider (CSP), you will be directed to contact that provider.

The issue is as a CSP, Communications Credits is not available to sell to customers. Feel free to vote here: https://www.skypefeedback.com/forums/299913-generally-available/suggestions/32347861-add-communication-credits-in-the-csp-program. As a CSP Customer, you’ll be directed to use the Office 365 Admin Center and source the Communications Credits under “Add-on subscriptions”.

The problem, if the base user subscription (E#, Skype for Business Online Plan 2, Microsoft Phone System and/or Audio Conferencing license) was purchased from a CSP, the Communications Credits add-on is not listed to purchase directly from Microsoft.

This exposes an interesting licensing dilemma, add-ons must be purchased the same way as the base license required for the add-on. Example if consuming E3 Trials you will not be able to purchase a Microsoft Phone System add-on, as the add-on must also be in a trial state. If you purchased E3 from one CSP or direct from Microsoft, the Microsoft Phone System add-on can’t be purchased from a different CSP.

There is a workaround…. Add-ons do not have a 1:1 check with the base license, you could have 1 x E3 and 1000 x Microsoft Phone System add-ons.

If you contact Microsoft Support for adding Communications Credits to your account when sourcing licensing from a CSP, you will be told to: Provision an Office 365 Enterprise E5 through Advisor Platform or Direct from Microsoft, once provisioned, add-on for Communication Credits will be available, once add-on is available, make sure to assign to all the users then cancel Office 365 Enterprise E5 within 24 hours. Any subscription canceled within 24 hours will not be charged through Advisor platform.

Once you have the Communications Credits add-on, you can determine the amount and if it auto-recharges.

Additional info: Set up Communications Credits for your organization