A few months back I was working on a Cisco Conductor implementation and ran into a bit of a hiccup. Here is what we set out to do:

• Conductor integrated to CUCM to provide ad-hoc conference resources for Jabber and TelePresence endpoints with the following versions & requirements:
  • CUCM 10.5
  • Conductor 3.0
  • vTelePresence Server 4.0
  • All media and signaling must be encrypted
  • Symantec managed PKI for internal server certs

Everything worked fine using unencrypted traffic. In fact, it even worked fine when the media was encrypted and the signaling was left unencrypted. To be even more specific, we could even encrypt half of the signaling successfully, but getting all of the signaling encrypted posed a problem. What’s even worse, a cosmetic bug led to some tail-chasing. Let’s take a look at the integration between CUCM and Conductor and break things down.

The config guides state that two items must be configured on CUCM.

1. A SIP Trunk must be created. This is how CUCM sends calls to the Conductor and ultimately to the TelePresence Servers (TS) it represents. This is done via SIP signaling and RTP media.
2. A Conference Bridge Media Resource. This is what CUCM uses to make API calls via HTTP(S) to Conductor to create those ad-hoc video conferences on demand. The API call happens before CUCM sends the SIP signaling for the actual video call so that Conductor creates the conference and knows where to send that incoming SIP INVITE and video call.

Getting this all to work unecrypted was no problem, so let’s look at how we control encryption for these three items:

1. API signaling (TCP port 443, HTTPS)
2. SIP signaling (TCP port 5061, SIPS)
3. RTP media (UDP ports 16384-32767, sRTP)

The SIP Trunk controls the SIP Signaling and the Media. On the SIP Trunk configuration page the first setting is to check the SRTP allowed box.

The next item is to specify the SIP signaling port as 5061 as that is the port where Conductor expects to receive secure SIP signaling. We should also set a custom SIP Trunk Security Profile. More on that next.

The SIP Trunk Security Profile allows us to define a few things that force the encryption.

1. Set the device security mode to Encrypted to force encryption.
2. Set the expected transport both inbound and outbound to force TLS.
3. Set the expected inbound port (listening port) to 5061 (TCP).
4. The item I skipped over is the X.509 Subject Name. This is what CUCM uses to validate the certificate it receives from Conductor. It will look at the received certificates Subject Name as well as any SANs for a match against this value. If there is a match (and the certificate has a valid, trusted chain) the certificate validation will succeed. Note that this field is case sensitive.

At this point it is important to note that this is as specified in the deployment guide for Conductor/CUCM 3.0 even though these settings are what I would call “bad practice”. You know, the opposite of best practice. The config guide has one easy no-no to pick out (my screenshot is “corrected”). They use a hostname that is not fully qualified in the X.509 Subject Name field of the Security Profile, which we know makes our security friends cry themselves to sleep at night. The second seems a little more innocuous at first. They state you should enter an IP address rather than FQDN on the SIP Trunk config page’s Destination Address setting. This is particularly interesting because it shows that CUCM can have a secure SIP Trunk pointed at an IP address, but succeed the certificate validation because it will use the value in the SIP Trunk Sec Profile’s X.509 Subject Name to match an FQDN from the Conductor’s certificate.

Why are these “bad practice”? Because the CAB Forum says that only public FQDNs should be used in certificates. See this note from the CAB Forum: Guidance on Internal Names (and IP addresses).

… on November 11, 2015, the issuance of certificates with a reserved IP address or internal server name is prohibited.  On October 1, 2016, all publicly trusted SSL/TLS certificates with an internal name or reserved IP address will be revoked and/or blocked by browser software.  As of July 1, 2012, all CAs were required to notify customers applying for internal name certificates that the use of such certificates has been deprecated by the CA / Browser Forum and that the practice will be eliminated by October 2016.

Let’s keep moving with our Conductor config, but… spoiler alert: this will end up being the crux of our issue.

I should point out that at this point if you have properly installed certificates on both CUCM and Conductor including all necessary root and intermediate CA certs to the appropriate trust stores that this SIP Trunk will be ‘operational’. If you enable SIP Options Pings in CUCM 10.5 the status of the SIP Trunk will show “up” in the CCMadmin interface. We’ve knocked out 2/3 of the config. The SIP signaling and RTP (sRTP in this case) are successfully configured.

The last piece is the API integration via the CUCM Conference Bridge media resource. Let’s do that. Under Media Resources > Conference Bridge we create a new CFB of type Cisco TelePresence Conductor similar to this ensuring the Use HTTPS box is checked.

And this is where it all falls apart. The CFB will never register. Well.. sort of never. Time for an aside on the cosmetic bug, CSCuo85081.

The bug notes, as with most Cisco bugs, are a bit confusing. The net of this is that if, when you create this CFB, you leave Use HTTPS unchecked and CUCM successfully creates an unencrypted (HTTP) connection to Conductor and you then check the box Use HTTPS it will not actually use HTTPS. Even after multiple Resets, Apply Configs, etc. the existing HTTP sessions are not torn down nor are new HTTPS sessions created. You must reboot UCM or delete the CFB and recreate it checking the Use HTTPS box upon creation in order for HTTPS to be used. Imagine the fun had tracking this down via packet captures and mysterious CFB “un”-registrations the morning after a cluster reboot. It really is true, packet captures don’t lie.

Perhaps the worst part is that the box Use HTTPS will stay checked, the endpoints will report they are connected securely to Conductor and everyone assumes life is good. Until you reboot and it all breaks. Or when the security team audits the traffic and sends you directly to jail, do not pass go, do not receive $200 when they find your Conductor admin credentials being sent in clear text via this HTTP call from CUCM. By the way, to mitigate you could(should) create a new user on Conductor with API permissions only (no admin/GUI rights) to use for your CUCM integration.

Okay, so we know to check our packet captures to ensure our traffic is truly encrypting before declaring victory. So why won’t HTTPS work? Remember back to our SIP Trunk and SIP Trunk Sec Profile when I noted the interesting behavior that the destination was an IP address, but the certificate validation was done on a different field? The CFB traffic does not benefit from this ability to specify two different addresses. The destination address of the SIP Trunk that is specified on the Conductor CFB config page is used both as the destination of the API call and as the subject name to validate against the received certificate.

Remember that we configured an IP address as the destination of the SIP Trunk. That IP address does not show up as the Subject Name on any of the SANs so the certificate validation fails and the HTTPS connection cannot be established.

Okay, so let’s just use that next checkbox, the one that says Override SIP Trunk Destination as HTTP Address.

Ah-ha! Success! It validates successfully against one of our certificate’s SANs and registers. Packet cap confirms the traffic is encrypted. Life is good. Let’s place a test call.

Hrm… call failed trying to conference in the third endpoint.

Pro tip: If you ever find yourself needing to look at Conductor diag logs… don’t. It’s not that you can’t find useful info in them, but they are obnoxiously verbose and difficult to read mostly due to the use of line endings every fifth character.

The Conductor logs tell us that the conference.create step failed because Conductor couldn’t find a valid Location for the IP address in the API call. The logs show the FQDN being used as we specified in our “Override SIP Trunk Destination as HTTP Address” setting to get the CFB to pass the cert validation and register. But it says it can’t find a Location for the IP Address. Ah, here is our next problem. Conductor logic for matching inbound calls to conference templates is driven by the IP address assigned on the Conductor Location page. This is why you must create all those ‘secondary’ IP addresses on Conductor. Each new conference template must be assigned its own IP address. At this time, Conductor will only match against IP and does not have any logic to match the corresponding FQDN.

Well… that puts us in a bit of a pickle. If we tell CUCM to use the IP address for this API call the cert validation fails. And if we tell it to use the FQDN the cert validation succeeds, but the API call fails because Conductor doesn’t understand FQDNs. This is why the deployment guide states that IP addresses MUST be used for the SIP Trunk to Conductor and that the “Override destination” setting should not be used on the CFB config page.

Apparently most of Cisco’s customers are okay with sending admin/API credentials in clear text across their internal networks as I was surprised to find TAC and, via escalations, the BU had not come across this before. Their recommendation is to include the IP addresses of the Conductor (yes, all of them) as SANs in the certificate so that the certificate validation succeeds when telling CUCM to connect to the Conductor IP addr rather than FQDN.

That’s all fine and good if you are signing certificates via an internal CA, but if using a public CA or a managed PKI service from someone like Symantec for your internal PKI you will be hard-pressed to find anyone willing to sign a CSR that includes an IP address. See the CAB Forum Guidance on Internal Names discussion earlier in this post. I ended up having to generate a self-signed certificate and install the self-signed cert directly to CUCM’s Tomcat trust store.

But here we get to the last fly in the ointment so to speak. You cannot use the GUI to generate this CSR. You must use OpenSSL to generate a CSR off-box with the correct parameters. Why? Because the GUI generates a CSR with all SANs set to a type of DNS. CUCM will not validate the certificate properly unless the IP addresses are set to a type of IP address. See the difference in the two below.

Using the Conductor GUI to generate a CSR yields the following:

DNS:conductor.widgets.co
DNS:conductor-adhoc.widgets.co
DNS:conductor-rendezvous.widgets.co
<strong>DNS</strong>:10.10.10.10
<strong>DNS</strong>:10.10.10.11
<strong>DNS</strong>:10.10.10.12

With a certificate created using SANs as above the cert validation still fails. What we need is this:

DNS:conductor.widgets.co
DNS:conductor-adhoc.widgets.co
DNS:conductor-rendezvous.widgets.co
<strong>IP Address</strong>:10.10.10.10
<strong>IP Address</strong>:10.10.10.11
<strong>IP Address</strong>:10.10.10.12

We cannot do this via the Conductor GUI because it does not allow us to specify the type of SAN. If you are familiar with Expressway you will note that its CSR generation tool does give you some flexibility here specifying SANs of type DNS or XMPP, etc. We would need something similar in Conductor, but it’s just not there. Instead we need to generate a certificate off-box using OpenSSL. You can take the existing private key from Conductor and re-use it to create the new certificate or you can generate a new private key. The command below was run from Terminal on a Mac and generates a new private key and cert.

openssl req -x509 -nodes -days 1095 -newkey rsa:2048 -keyout newprivkey.key -out conductor.pem -config openssl.cnf

What do we have here? We tell openssl to generate 2048-bit RSA encrypted private key named ‘newprivkey.key’. We also tell it to generate a certificate named ‘conductor.pem’ using a special config named openssl.cnf.  The modified openssl.cnf file should be in the same directory where you are running this command and where the private key and CSR will be created. What needs to be in that openssl.cnf file? First, let’s find the default config file.

ben@vw$ locate openssl.cnf
/System/Library/OpenSSL/openssl.cnf

Copy that to some new directory and we can make some edits.

$cp /System/Library/OpenSSL/openssl.cnf ~/scratch/conductorcert/

First, find the [req] section and change the default bits to 2048.

####################################################################
[ req ]
default_bits = <strong>2048</strong>

Since this is a self-signed cert and we are not actually generating a CSR (where we would use the [v3_req] section) we use the [ v3_ca ] section to add keyUsage and SAN information. Normally we’d specify things such as dataEncipherment for keyUsage as well as serverAuth, clientAuth and IPsec End System to the extendedKeyUsage parameters. Here we are going to leave these commented out. If no parameters are set here then the certificate can be used for any of them. We just leave CA:FALSE set. The keyUsage and extendedKeyUsage parameters you see here are commented out, but include the values you would normally want in a CSR for Conductor. Add dataEncipherment and include serverAuth, clientAuth and IP Sec End System. When I tried setting these I had some issues getting CUCM to accept the certificate. Admittedly, I probably could have looked a little harder at this to find the right combo of extensions to make this work with a self-signed cert, but sometimes you just need to take off and nuke it from orbit. At this point we’re breaking so many security best practices with this self-signed cert I wasn’t too concerned. Plus, in theory this is temporary as Cisco is supposed to provide a permanent fix in an upcoming release (see notes at the end for the bug to track if you’re interested). Finally, we include a line telling it to add an array of alternate names to this certificate.

[ v3_ca ]

basicConstraints = CA:FALSE
#keyUsage = nonRepudiation, digitalSignature, keyEncipherment, dataEncipherment
#extendedKeyUsage=serverAuth,clientAuth,1.3.6.1.5.5.7.3.5
subjectAltName = @alternate_names

Next we will create our array of alternate names (this section does not exist in the default file, we are creating a new section named alternate_names). Remember that this must include every FQDN and every IP address used by this Conductor. The admin FQDN/IP and any others used for ad-hoc or rendezvous conferences. Add to this list as needed for your install.

[ alternate_names ]
DNS.0 = conductor.widgets.co
DNS.1 = conductor-adhoc.widgets.co
DNS.3 = conductor-rendezvous.widgets.co
IP.0 = 10.10.10.10
IP.1 = 10.10.10.11
IP.2 = 10.10.10.12

We are now ready to generate our private key and self-signed certificate. Referencing again the same command as above, this will create those two items with our modified openssl.cnf file.

ben@vw$openssl req -x509 -nodes -days 1095 -newkey rsa:2048 -keyout newprivkey.key -out conductor.pem -config openssl.cnf
Generating a 2048 bit RSA private key
.................................+++
...................................+++
writing new private key to 'newprivkey.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Minnesota
Locality Name (eg, city) []:Minneapolis
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Polzin
Organizational Unit Name (eg, section) []:Lab
Common Name (e.g. server FQDN or YOUR name) []:conductor.widgets.co
Email Address []:collab@widgets.co

Let’s take a look.

ben@vw$ls -la
total 40
drwxr-xr-x@  5 ben  staff    170 May 25 13:07 .
drwxr-xr-x@ 22 ben  staff    748 May 25 13:05 ..
-rw-r--r--@  1 ben  staff   1976 May 25 13:07 conductor.csr
-rw-r--r--@  1 ben  staff   1675 May 25 13:07 newprivkey.key
-rw-r--r--@  1 ben  staff  10093 Feb 11 23:09 openssl.cnf

We now have a private key and certificate, let’s check that cert.

ben@vw$openssl x509 -in conductor.csr -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            b6:de:10:3a:38:17:d2:0c
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: C=US, ST=Minnesota, L=Minneapolis, O=Polzin, OU=Lab, CN=collab@widgets.co/emailAddress=collab@widgets.co
        Validity
            Not Before: May 25 18:21:48 2015 GMT
            Not After : May 24 18:21:48 2018 GMT
        Subject: C=US, ST=Minnesota, L=Minneapolis, O=Polzin, OU=Lab, CN=collab@widgets.co/emailAddress=collab@widgets.co
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
            RSA Public Key: (2048 bit)
                Modulus (2048 bit):
                    00:a4:2f:66:aa:0c:fc:0d:2b:32:b6:5b:18:51:65:
                    ab:29:66:80:cf:73:7d:3e:d1:a5:a2:9e:a2:f0:41:
                    51:0d:70:9e:7d:64:a9:97:e2:6a:7d:4e:ab:3f:79:
                    33:11:74:98:83:9d:e9:a9:ab:14:eb:26:db:ae:1e:
                    31:5a:d2:95:22:d2:88:b3:88:09:27:73:73:4a:ca:
                    5d:35:b9:69:0d:f6:bd:b6:d6:7e:a7:47:46:40:17:
                    70:6b:86:45:11:33:43:49:7c:7b:7b:ad:7c:80:3e:
                    ee:5b:14:03:bc:8d:71:73:21:a1:0d:fd:48:0a:69:
                    60:c1:37:be:75:98:a9:36:9a:84:94:15:93:53:17:
                    11:cc:d0:dd:78:e2:f8:bb:40:ba:af:5c:43:ac:25:
                    18:57:1c:64:96:c8:1c:30:27:a0:18:36:18:e6:6c:
                    83:e3:44:58:29:fa:02:9f:d2:ea:2b:0e:45:6d:f1:
                    29:31:bc:9a:5d:34:b9:a0:dc:55:de:a3:65:b5:f9:
                    ad:62:95:03:74:65:4e:6d:d2:a3:56:45:2c:fb:c1:
                    e8:de:17:2d:c0:11:81:c1:13:20:b1:f1:84:47:26:
                    3b:7e:4e:53:c0:c8:e1:8a:e7:12:91:64:e2:06:41:
                    ae:d5:02:de:e9:6a:57:d5:b0:63:c8:53:1a:64:47:
                    9d:dd
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints: 
                CA:FALSE
            X509v3 Subject Alternative Name: 
                DNS:conductor01.widgets.co, DNS:conductor-adhoc.widgets.co, DNS:conductor-rendezvous.widgets.co, IP Address:10.10.10.10, IP Address:10.10.10.11, IP Address:10.10.10.12
            X509v3 Subject Key Identifier: 
                3C:32:14:CC:D0:C4:79:5E:C1:28:25:16:43:89:3C:8C:CD:AB:2F:2C
            X509v3 Authority Key Identifier: 
                keyid:3C:32:14:CC:D0:C4:79:5E:C1:28:25:16:43:89:3C:8C:CD:AB:2F:2C
                DirName:/C=US/ST=Minnesota/L=Minneapolis/O=Polzin/OU=Lab/CN=collab@widgets.co/emailAddress=collab@widgets.co
                serial:B6:DE:10:3A:38:17:D2:0C

    Signature Algorithm: sha1WithRSAEncryption
        86:fe:7d:1e:d8:97:76:52:9a:49:fd:82:1b:bd:fa:05:80:87:
        12:bb:4a:a6:97:44:9d:eb:52:e5:58:63:99:8d:66:bd:ab:82:
        b5:01:6e:ce:52:24:77:61:21:a2:d4:a0:dd:c9:f7:30:2d:7f:
        be:bb:73:22:6b:82:53:b8:14:90:5d:ba:ae:39:da:6e:1e:fe:
        8a:a8:08:17:bb:56:29:2a:da:95:16:3e:f2:46:e6:84:62:88:
        41:1c:8c:38:58:07:b6:b8:b9:72:10:ca:3f:7d:0e:d5:c1:ea:
        40:b5:0f:9a:fa:c9:53:48:94:52:68:e8:9d:f7:6a:be:68:d5:
        3f:a9:3f:bd:80:c9:ab:17:f4:9c:15:c1:e4:1e:85:68:3d:5c:
        ec:42:24:f4:bb:a4:fe:4a:b8:d1:2d:e2:a5:8c:f5:8d:5e:e5:
        af:25:26:e6:3d:40:f7:f4:61:61:b4:5f:89:da:1f:02:8a:a6:
        53:61:23:52:30:ed:a4:ce:a8:6c:69:7b:ba:65:b2:de:28:18:
        3d:fd:15:a3:e6:bf:7a:41:f2:62:a7:90:71:bf:f3:ac:c5:81:
        22:ab:43:8a:8b:fa:cc:63:fd:fd:f7:fe:8e:bf:7c:02:1a:40:
        97:86:fc:00:9b:13:90:f7:32:36:7c:b4:17:c0:51:63:1d:45:
        f1:f2:46:86

Now to get this installed on Conductor.

SSH to Conductor as root and navigate to /tandberg/persistent/certs.

ben@vw$ssh root@conductor.widgets.co
Password:
Last login: Tue May  3 18:59:01 CST 2015 from 10.10.10.1 on pts/1
~ # cd /tandberg/persistent/certs

Copy the existing private key and certificate in case you need to fall back (note that the SSH session uses a different certificate so making modifications here will not kill you SSH session or ability to connect to the Conductor).

~/persistent/certs #cp privkey.pem privkey.pem.bak
~/persistent/certs #cp server.pem server.pem.bak

Use SCP to transfer the new private key and cert or use vi and copy paste. For variety, SCP for private key and vi for cert. Use either one.

~/persistent/certs # scp ben@10.10.10.1://Users/ben/scratch/newprivkey.pem ./privkey.pem
The authenticity of host '10.10.10.1 (10.10.10.1)' can't be established.
RSA key fingerprint is 77:ff:dd:55:22:dd:44:aa:77:bb:66:11:22:33:44:55.
Are you sure you want to continue connecting (yes/no)? yes
Password:
conductor.csr                                               100% 1830     1.8KB/s   00:00   

And the cert (again, feel free to use SCP, but if not available you can always copy/paste into your SSH session using vi).

~/persistent/certs # touch selfsignedconductor.pem
~/persistent/certs # vi selfsignedconductor.pem
...
Open the cert on your machine with a text editor, copy the entire contents.
Back in vi on Conductor:
type i for insert mode
type Ctrl-V for paste
type :wq to save and quit
...
~/persistent/certs #cp selfsignedconductor.pem server.pem

Reboot Conductor. Validate the certificate is as expected by viewing it via a browser. The list of SANs must look like this with special attention to the IP address: type.

DNS:conductor.widgets.co
DNS:conductor-adhoc.widgets.co
DNS:conductor-rendezvous.widgets.co
IP Address:10.10.10.10
IP Address:10.10.10.11
IP Address:10.10.10.12

Now take that same certificate file and upload it to CUCM’s tomcat-trust certificate store on all nodes. Restart Tomcat on all nodes. CUCM will now properly validate the certificate even when using the IP address as the destination of the SIP Trunk and Conductor CFB object in CUCM. And because we used the IP address the API calls will succeed because Conductor is able to match those IP addresses to Locations and their assigned IP addresses.

I expect that Cisco will provide some workaround in a future release of Conductor.

There is a relatively easy workaround that they could implement by allowing you to specify the SAN type when creating the CSR in Conductor to specify a SAN being an IP address rather than DNS name (update 5/28/2015, this has been assigned CSCuu53177). This would be a poor workaround as using IP addresses in certificates is frowned upon and will be deprecated next year by all CAs (most will not issue certs with IP addresses today).

The real fix will be to modify the logic of Conductor to match Locations to IP addresses AND their associated FQDNs. In this way we could do all of our config in CUCM by FQDN and still have those API calls succeed during that conference.create step as Conductor matches against its Locations.

Related Cisco Bug IDs: CSCut22572, CSCuq90392