One of the critical stages in implementing and scaling your SD-WAN is the onboarding of WAN Edge devices. During this process, network administrators may occasionally encounter certificate signing issues between WAN Edge routers and vManager. One common root cause of such errors is a time synchronization mismatch between the WAN Edge device and vManager.

In this article, we’ll explore:

• Understanding the role of certificates in SD-WAN onboarding

• Common symptoms of certificate errors

• How to troubleshoot time synchronization issues between vManager and WAN Edge devices

• Steps to fix the issue and prevent it from recurring

1. Understanding the Role of Certificates in Cisco SD-WAN

In Cisco Catalyst SD-WAN, secure communication between the control plane (vManager, Controller, and Validator) and the WAN Edge devices (routers) is critical. This is achieved through Public Key Infrastructure (PKI)-based certificates. Each WAN Edge device must have a valid certificate, which is signed by vManager, to authenticate itself and join the network securely.

The certificate exchange process ensures that:

• WAN Edge devices are trusted and can participate in the SD-WAN fabric.

• The integrity and confidentiality of the data flowing between network nodes are protected.

When onboarding a WAN Edge device, it communicates with vManager to retrieve and install the signed certificate. However, if there is a mismatch in system time between the devices, this exchange can fail, leading to errors.

2. Common Symptoms of WAN Edge Certificate Errors

One of the primary indicators of a certificate signing issue is when vManager fails to onboard the WAN Edge device. You may see error messages in the vManager dashboard or logs such as:

• "Certificate Installed Failed"

• "Certificate is invalid or expired"

These issues often arise when the time on the WAN Edge device is not properly synchronized with the vManager server. PKI systems rely heavily on time synchronization to validate certificates, especially regarding expiration dates and validity periods. Even a slight time difference can lead to certificate rejection.

Check detailed logs via vManager shell to see insightful errors

Manager-20-9# vshell

Manager-20-9:~$ tail -f /var/log/nms/vmanage-syslog.log

---SNIP---

local6.INFO : 09-Oct-2024 02:11:34,762 UTC INFO  [] [VmanageSyslogLogger] (Thread-239) ||

      vedge-cloud: {"logid":"8598f1c7-0f4c-4f02-a5e6-96bd48de43e1","entry_time":1728439894761,"statcycletime":1728439894761,"logmodule":"vedge-cloud","logfeature":"vedge-cloud","loguser":"system","logusersrcip":"90.90.90.1",

      "logmessage":"Certificate Installation failed on vEdge cloud by vManage-b5657745-df80-4669-97ae-6a7b96890f3e-C8K-57282144-9F31-3AF3-799E-F33FDEFEF744","logdeviceid":"10.0.1.1",

            "auditdetails":["Failed to install certificate signed by vmanage <signerVmanageUUID: b5657745-df80-4669-97ae-6a7b96890f3e>

            on vedge <uuid: C8K-57282144-9F31-3AF3-799E-F33FDEFEF744, systemIP: 10.0.1.1>.

      Vedge cloud clock <1728439894000 ms> and vmanage clock <1728462976000 ms> are off by 23082000 ms"],"logprocessid":"436903f6-cefd-4c1e-8fd0-9f2578c5ea8c"}

---SNIP---

3. Troubleshooting Time Synchronization Between vManager and WAN Edge

To resolve the certificate error, it is essential to investigate whether time synchronization between the WAN Edge device and vManager is out of sync.

Step 1: Check Time on vManager and WAN Edge

Start by verifying the current time on both the WAN Edge device and the vManager server. You can check the time on these devices using the following commands:

For vManager:

show clock

For WAN Edge:

show clock

If there is a noticeable difference in time between the two devices, it will likely cause certificate validation failures.

Step 2: Verify NTP Configuration

Time synchronization in network environments is typically managed through the Network Time Protocol (NTP). To ensure the time is in sync, both the vManager server and WAN Edge device should be correctly configured to synchronize with the same NTP server.

For vManager NTP status:

Manager-20-9# show ntp associations

IDX  ASSOCID  STATUS  CONF  REACHABILITY  AUTH  CONDITION  LAST EVENT   COUNT

-------------------------------------------------------------------------------

1    15607    8013    yes   no            none  reject     unreachable  1

2    15608    961a    yes   yes           none  sys.peer   sys_peer     1

Manager-20-9# show ntp peer



INDEX  REMOTE  REFID   ST  TYPE  WHEN  POLL  REACH  DELAY   OFFSET  JITTER

--------------------------------------------------------------------------------------------

1    127.127.1.0  .LOCL.    5   l   105m  64   0    0.000   +0.000  0.000

2   *103.130.217.41  162.159.200.1  4   u     174   256   377    11.435  -0.179  0.300

For WAN Edge NTP status:

S1001-SD1#show ntp associations

  address         ref clock       st   when   poll reach  delay  offset   disp

 ~11.10.1.254    115.165.161.155  3      6     64     3  2.000  36.000 1939.2

 * sys.peer, # selected, + candidate, - outlyer, x falseticker, ~ configured

Ensure that both devices point to the same NTP server and that the server itself is functioning properly.

Step 3: Check for Time Drift

Time drift is another common issue, especially if devices have been running for a long period without synchronization. This can occur if the NTP configuration is incorrect or if the devices lose connectivity to the NTP server. You can manually update the time on the devices temporarily while investigating the root cause of the NTP issue.

Step 4: Correct Time Settings

If you notice a time mismatch, correct the settings by configuring NTP on both the vManager and the WAN Edge device. Below is an example of how to configure NTP on vManager and WAN Edge devices:

For vManager NTP configuration:

system server 1.vn.pool.ntp.orgvpn 512 version 4 exit !

For WAN Edge NTP configuration:

! ntp server 11.10.1.254 prefer !

Choosing your local NTP server or Regional Internet NTP server.

Note, don’t forget to allow ntp service on the sdwan tunnel interface that communicates with ntp server (if you are using public or outside ntp server).

sdwan interface GigabitEthernet2 tunnel-interface allow-service ntp ! !

Step 5: Restart Certificate Process

Once the time is synchronized correctly across vManager and WAN Edge, restart the certificate signing process via vManager. This will allow vManager to reissue a valid certificate and complete the onboarding.

Via vManager GUI:

Navigate to Configuration > Certificates > Select target WAN Edge by Chassis ID > Renew Device CSR

4. Preventing Future Time Sync Issues

After fixing the issue, it’s essential to implement preventive measures to avoid future time synchronization problems:

• Regular Monitoring of NTP Status: Ensure that NTP configurations on both vManager and WAN Edge devices are regularly monitored for status and health.

• Automate Alerts for Time Drift: Set up alerts in vManager to notify administrators if there is a significant time drift between the WAN Edge and vManager.

• Redundant NTP Servers: Configure multiple NTP servers to ensure time synchronization even if one NTP server goes down.

• Periodic Clock Verification: Perform routine checks on clock settings, particularly after maintenance or software updates, to ensure accurate time synchronization.

Summary

Time synchronization plays a vital role in network management, but it’s easy to overlook, especially in Cisco Catalyst SD-WAN environments. If your WAN Edge device is having trouble with certificate signing during onboarding, it’s a good idea to check the time settings on both vManager and the device. A difference in time between them can cause certificate validation to fail and interrupt the onboarding process.

By understanding common issues like time drift or incorrect NTP configurations, and following best practices, you can troubleshoot and fix these problems quickly. Keeping an eye on time settings across your SD-WAN network will help maintain smooth operations and prevent certificate errors down the road.