CUCM 12.5 CDR billing and DRS server failure after upgrade

 Tl;dr: CUCM 12 and beyond supports a shorter list of encryption algorithms for SFTP file transfers.

After completing a CUCM cluster upgrade from 11.5(1)SU6 to 12.5(1)SU3 I found errors in RTMT indicating CDR records were no longer being sent to the existing RSI billing server.

There were no networking changes and all CUCM services were running so we started with restarting the Cisco CDR Repository Manager and Cisco CDR Agent because who knows, why not?  When that failed I quickly added another billing server and pointed it to a freeFTPd server I was using to serve Cisco upgrade media.  I immediately started receiving billing records there. With that there then didn't seem to be any issue with the CUCM record collection and transfer themselves but something unique to the old billing server.

Collect logs from CUCM publisher

I grabbed all the logs related to CDR from the Publisher via RTMT, used grepWin to dig through them to find references to the billing server IP address configured in CUCM and found: 

2020-12-29 05:47:57,549 INFO  [Thread-15] cdrrep.CDRSender (CDRSender.java:175) - There was file delivery failure for destination 2 in last round, re-connect server now!

2020-12-29 05:47:59,619 ERROR [Thread-15] sftpapi.SFTPConnection (SFTPConnection.java:319) - error Making SFTP connectionAlgorithm negotiation fail

2020-12-29 05:47:59,619 ERROR [Thread-15] sftpapi.ftpClient (ftpClient.java:246) - connect(): Failed connect to 10.10.10.10

The interesting bit there is Algorithm negotiation fail.

Some Googling turns up a bug CSCuz80145 indicating I may need to eliminate some CTR ciphers:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz80145

Of course my version is well beyond the affected and fixed versions so how it applies is questionable.  It does maybe mean there is some sort of issue with using the CTR ciphers.

The Administration Guide for Cisco Unified Communications Manager, Release 12.5(1)SU1 (which I honestly have never looked at) does indeed indicate that ONLY the CTR ciphers are supported in version 12 and beyond, at least for DRS purposes.

aes128-ctr, aes192-ctr and aes256-ctr supported

The RSI support representative the client contacted was in no mood to answer questions about what SFTP server was installed or how it was configured and simply suggested it was a CUCM issue. Cisco TAC was not going to respond anytime soon to confirm whether CDR transport had the same limitations as DRS.

How does one confirm what the SFTP servers support without their vendor support?

I fell upon a post by Kenneth Perry at https://nocthoughts.wordpress.com/2019/07/25/cdr-export-to-billing-server-failing/ describing a very similar issue discovered in a different fashion.  Given the date on his post and his SFTP server scans it seems he was working on an older CUCM version that did not support CTR ciphers, while my 12.5 version ONLY supported CTR ciphers.  Regardless, thanks Ken and whoever your colleague Mark is!

Although not my normal go-to tool, I did happen to have nmap installed on my laptop to prep for another "Cisco DevNet, DevOPs, CI/CD, automate everything and move it to the cloud training bootcamp" I sat through recently.

More info on the specific nmap script used and nmap in general is here: https://nmap.org/nsedoc/scripts/ssh2-enum-algos.html

Scanning the failing RSI as well as my successful test SFTP server returned what I suspected.  The old RSI billing server was not advertising the new CTR cipher support that the 12.5 admin guide indicated is required. The questionable but free and easy freeFTPd server showed it supported the CTR ciphers required by 12.5 as well as the old CBC list used by 11.5.

cbc only list on failing server

What to do?

RSI support did ultimately attempt to update their SFTP server to support the new ciphers but it didn't make a difference.  

Ultimately we put our fingers in our ears, flipped it all to FTP and called it a day.

Pretty cool regardless.

Update 02242022

Was recently tasked with configuring OpenSSH on a Windows Server 2016 (before it was offered as an installable Windows app / feature).
It was to serve as the target of multiple version 11.5/6 UC servers.

Notable configuration options used:

# Ciphers and keying specifically for 11.X but not newer
Ciphers aes128-cbc
KexAlgorithms diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

# To restrict access to a particular Windows local group, here called SFTPUsers
AllowGroups SFTPUsers