CUCM 12.5 CDR billing and DRS server failure after upgrade

 Tl;dr: CUCM 12 and beyond supports a shorter list of encryption algorithms for SFTP file transfers.

After completing a CUCM cluster upgrade from 11.5(1)SU6 to 12.5(1)SU3 I found errors in RTMT indicating CDR records were no longer being sent to the existing RSI billing server.

There were no networking changes and all CUCM services were running so we started with restarting the Cisco CDR Repository Manager and Cisco CDR Agent because who knows, why not?  When that failed I quickly added another billing server and pointed it to a freeFTPd server I was using to serve Cisco upgrade media.  I immediately started receiving billing records there. With that there then didn't seem to be any issue with the CUCM record collection and transfer themselves but something unique to the old billing server.

Collect logs from CUCM publisher

I grabbed all the logs related to CDR from the Publisher via RTMT, used grepWin to dig through them to find references to the billing server IP address configured in CUCM and found: 

2020-12-29 05:47:57,549 INFO  [Thread-15] cdrrep.CDRSender (CDRSender.java:175) - There was file delivery failure for destination 2 in last round, re-connect server now!

2020-12-29 05:47:59,619 ERROR [Thread-15] sftpapi.SFTPConnection (SFTPConnection.java:319) - error Making SFTP connectionAlgorithm negotiation fail

2020-12-29 05:47:59,619 ERROR [Thread-15] sftpapi.ftpClient (ftpClient.java:246) - connect(): Failed connect to 10.10.10.10

The interesting bit there is Algorithm negotiation fail.

Some Googling turns up a bug CSCuz80145 indicating I may need to eliminate some CTR ciphers:

https://bst.cloudapps.cisco.com/bugsearch/bug/CSCuz80145

Of course my version is well beyond the affected and fixed versions so how it applies is questionable.  It does maybe mean there is some sort of issue with using the CTR ciphers.

The Administration Guide for Cisco Unified Communications Manager, Release 12.5(1)SU1 (which I honestly have never looked at) does indeed indicate that ONLY the CTR ciphers are supported in version 12 and beyond, at least for DRS purposes.

aes128-ctr, aes192-ctr and aes256-ctr supported

The RSI support representative the client contacted was in no mood to answer questions about what SFTP server was installed or how it was configured and simply suggested it was a CUCM issue. Cisco TAC was not going to respond anytime soon to confirm whether CDR transport had the same limitations as DRS.

How does one confirm what the SFTP servers support without their vendor support?

I fell upon a post by Kenneth Perry at https://nocthoughts.wordpress.com/2019/07/25/cdr-export-to-billing-server-failing/ describing a very similar issue discovered in a different fashion.  Given the date on his post and his SFTP server scans it seems he was working on an older CUCM version that did not support CTR ciphers, while my 12.5 version ONLY supported CTR ciphers.  Regardless, thanks Ken and whoever your colleague Mark is!

Although not my normal go-to tool, I did happen to have nmap installed on my laptop to prep for another "Cisco DevNet, DevOPs, CI/CD, automate everything and move it to the cloud training bootcamp" I sat through recently.

More info on the specific nmap script used and nmap in general is here: https://nmap.org/nsedoc/scripts/ssh2-enum-algos.html

Scanning the failing RSI as well as my successful test SFTP server returned what I suspected.  The old RSI billing server was not advertising the new CTR cipher support that the 12.5 admin guide indicated is required. The questionable but free and easy freeFTPd server showed it supported the CTR ciphers required by 12.5 as well as the old CBC list used by 11.5.

cbc only list on failing server

What to do?

RSI support did ultimately attempt to update their SFTP server to support the new ciphers but it didn't make a difference.  

Ultimately we put our fingers in our ears, flipped it all to FTP and called it a day.

Pretty cool regardless.

Update 02242022

Was recently tasked with configuring OpenSSH on a Windows Server 2016 (before it was offered as an installable Windows app / feature).
It was to serve as the target of multiple version 11.5/6 UC servers.

Notable configuration options used:

# Ciphers and keying specifically for 11.X but not newer
Ciphers aes128-cbc
KexAlgorithms diffie-hellman-group-exchange-sha1,diffie-hellman-group1-sha1

# To restrict access to a particular Windows local group, here called SFTPUsers
AllowGroups SFTPUsers

Cisco Unity Connection via MRA Expressway Core HTTP allow list

 Deploying Expressway version 12.6 to support MRA and B2B calling with CUCM 11.5(1)SU8 I found voicemail was unavailable via the Jabber clients when connected via Expressway.  

The clients would indicate they couldn't connect to VMREST via 443.

The Unity Connection servers were discovered in Expressway Core (Configuration > Unified Communications > Unity Connection Servers) and the inbound rules seemed to be automatically populated (Configuration > Unified Communications > HTTP Allow List > Automatic Inbound Rules).

Regardless I found "Access Denied" messages in the Expressway Core event logs stating otherwise.

The fix was to seemingly replicate the rules manually (Configuration > Unified Communications > HTTP Allow List > Editable Inbound Rules).

To keep the list simple I used prefixes to VMREST on port 443 and the root on port 7080.

Add SSL support to nginx deployments

This is essentially a copy and paste of important points found at https://www.techrepublic.com/article/how-to-enable-ssl-on-nginx/. Please stop and go there to get a more complete version.

If you are following the logic and nginx configuration steps found at http://webmaxtor.blogspot.com/2020/08/flask-python-sqlite-aws-ec2-nginx.html feel free to follow this abridged and custom version instead.

Generate a self-signed certificate:

sudo openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/nginx-selfsigned.key -out /etc/ssl/certs/nginx-selfsigned.crt

Configure nginx to use SSL:

sudo nano /etc/nginx/snippets/self-signed.conf

ssl_certificate /etc/ssl/certs/nginx-selfsigned.crt;
ssl_certificate_key /etc/ssl/private/nginx-selfsigned.key;

sudo nano /etc/nginx/snippets/ssl-params.conf

ssl_protocols TLSv1.2;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_ciphers
ECDHE-RSA-AES256-GCM-SHA512:DHE-RSA-AES256-GCM-SHA512:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384;
ssl_ecdh_curve secp384r1;
ssl_session_timeout 10m;
ssl_session_cache shared:SSL:10m;
ssl_session_tickets off;
resolver 8.8.8.8 8.8.4.4 valid=300s;
resolver_timeout 5s;
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";

Generate pem:

sudo openssl dhparam -out /etc/ssl/certs/dhparam.pem 2048

Assuming you are using the sites-enabled verses site-available, do the following. Alternatively you can do the same in site-available and create a link to it in site-enabled.

sudo nano /etc/nginx/sites-enabled/bttb

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    include snippets/self-signed.conf;
    include snippets/ssl-params.conf;

    server_name 100.25.168.210 therealsitename.com www.therealsitename.com;
    location /static {
        alias /home/ubuntu/bttb/app_BTTB/static;
    }

    location / {
        proxy_pass http://localhost:8000;
        include /etc/nginx/proxy_params;
        proxy_redirect off;
    }
}

server{
    listen 80;
    listen [::]:80;

    server_name 100.25.168.210 therealsitename.com www.therealsitename.com;

    return 302 https://$server_name$request_uri;
}

Check to see if nginx is SSL enabled

sudo ufw app list

Restart and try https: and http: access. Both should send you to your page over https but with a self-signed certificate.

sudo systemctl restart nginx

Flask Python SQLite AWS EC2 nginx gunicorn

Moving a Flask Python and SQLite application from a Windows desktop to Amazon EC2 instance running nginx and gunicorn

After adding a Webex Teams BOT function to a small team performance tracking application I created it was essentially a requirement to move it to a public server with a fixed IP address / DNS name. To keep it low cost and simple I chose to use a free-tier Amazon AWS EC2 instance running Ubuntu.  While Elastic Beanstalk is a free service AWS designed to simplify these deployments, the additional features deployed with it (S3, load balancing, etc) would only really complicate management.  After a  few failures using Elastic Beanstalk I also decided troubleshooting the deployment process wouldn't be a great use of time.  With that I deployed the server manually, configured the environment and moved the application manually.  Here are the bullet points, not unlike dozens of other tutorials out there:

I am not listing steps to deploy an AWS EC2 instance but here is a sample of what the production server looks like:

I used puTTY and WinSCP  to administer the server and transfer files. You can see see this blog post for a tip to allow file transfers to AWS servers using WinSCP.

The server configuration and application deployment steps were generally as follows:

• Create a requirements.txt file on the desktop to aid in Flask and python configuration on the target server:

                pip3 freeze > requirements.txt

• Decide on the install location o the target server. Here a directory in the default AWS EC2 user ubuntu home directory was used (/home/ubuntu/bttb)

                mkdir /bttb

• I SCP'd my files to the bttb folder using WinSCP here. See the picture above.
• update list of available packages:

                sudo apt update

• upgrade installed packages

                sudo apt upgrade

• install pip for python3

                sudo apt install python3-pip

• install venv to enable virtual environment use

                sudo apt install python3-venv

• create a virtual environment

                python3 -m venv bttb/venv

• activate the virtual environment

                cd bttb
                source venv/bin/activate

• install python requirements

                sudo pip3 install -r requirements.txt

• set environment variable to run Flask app and test

                export FLASK_APP=app_BTTB 

                flask run --host=0.0.0.0

• Flask should now be running the app at your AWS server IP address on port 5000 (default)
• Test web access and fix stuff if necessary.  Then stop Flask wit Ctrl-C.
• install nginx and gunicorn to replace Flask as the web framework
• nginx will be the webserver
• gunicorn will be running python code

                sudo apt install nginx

      sudo apt install gunicorn

• setup nginx and guicorn

                sudo rm /etc/nginx/sites-enabled/default
                sudo nano /etc/nginx/sites-enabled/bttb

• the nginx configuration file contents are like so:

server{

        listen 80;

        server_name 100.25.168.210;

        location /static {

                alias /home/ubuntu/bttb/app_BTTB/static;

        }

        location / {

                proxy_pass http://localhost:8000;

                include /etc/nginx/proxy_params;

                proxy_redirect off;

        }

}

• restart nginx

                sudo service nginx stop

                sudo service nginx start

                check the status with service nginx status

• you should now be able to view files in static folder, served by nginx
• via http://100.25.168.210/static/my_picture.png for example

• See https://serverfault.com/questions/947301/nginx-displaying-failed-13-permission-denied-when-trying-to-access-new-site and sudo -u www-data namei /home/ubuntu/bla/front/dist/vendor.js for issues nginx with permissions
• 
  
• the pages requiring python will fail now though
• test static file access and troubleshoot if required
• now, run gunicorn and specify 'workers' where worker is 2 x number of cores on server
• you should know this from the EC2 deployment or find it via 

                nproc --all

• here the create_app() represents a function in the __init__.py file in the app_BTTB directory

                gunicorn -w 3 'app_BTTB:create_app()'

• You should now have access to the python based pages as well as the static content
• test your application access and troubleshoot if required
• now install supervisor to start the application with the server and keep it running

                sudo apt install supervisor

• create a supervisor config file for your app

                sudo nano /etc/supervisor/conf.d/app_BTTB.conf

• the app_BTTB.conf contents are like so. Note you may need to reference the python virtual environment gunicorn path specifically:

[program:app_BTTB]

user=ubuntu

directory=/home/ubuntu/bttb

command=/home/ubuntu/bttb/venv/bin/gunicorn --chdir /home/ubuntu/bttb 'app_BTTB:create_app()' (note this line may appear word wrapped)

autostart=true

autorestart=true

stopasgroup=true

killasgroup=true

stderr_logfile=/var/log/app_BTTB/app_BTTB.err.log

stdout_logfile=/var/log/app_BTTB/app_BTTB.out.log

• create the supervisor log files referenced above

                sudo mkdir -p /var/log/app_BTTB
                sudo touch /var/log/app_BTTB/app_BTTB.err.log
                sudo touch /var/log/app_BTTB/app_BTTB.out.log

• reload supervisor

                sudo supervisorctl reload

                sudo supervisorctl status  

Assuming all goes as planned, your application is now running, accessible on the internet and running courtesy of nginx and gunicorn.

For reference, the tree of this application looks basically like below, where some folders were removed 

├── app_BTTB

│   ├── __init__.py

│   ├── auth.py

│   ├── db.py

│   ├── schema.sql

│   ├── static

│   │   ├── Boogie.png

│   │   └── Boogie_WebexTeams.png

│   ├── templates

│   │   ├── auth

│   │   ├── base.html

│   │   ├── tracker

│   │   └── webexteams

│   ├── tracker.py

│   └── webexteams.py

├── instance

│   ├── BTTB.sqlite

├── requirements.txt

└── venv

    ├── bin

    │   ├── activate

    │   ├── activate.csh

    │   ├── activate.fish

    │   ├── easy_install

    │   ├── easy_install-3.6

    │   ├── pip

    │   ├── pip3

    │   ├── pip3.6

    │   ├── python -> python3

    │   └── python3 -> /usr/bin/python3

    ├── include

    ├── lib

    │   └── python3.6

    ├── lib64 -> lib

    ├── pyvenv.cfg

    └── share

        └── python-wheels

Cisco 8845 undocumented speaker phone access feature

This was literally found in the wild by simply jiggling the table.  Gross.

Tony Hawk and I are friends now, so...