noVNC console disconnects when Jetty WebSocket reaches idle timeout on CloudStack 4.22.1.1

[montagnahuanghsiao]

problem

The noVNC console disconnects while the guest VM, Console Proxy VM,
CloudStack agent, network connectivity, and KVM VNC backend remain
operational.

The browser displays:

Failed to connect to server / access token has expired

The console disconnects even while the noVNC session is actively being used
with keyboard input and screen updates. This is not an inactive or abandoned
browser session.

Closing the old console window and opening a new console session from the
CloudStack UI generates a new access token and temporarily restores access.
The newly generated session eventually disconnects again.

Observed CPVM log messages include:

Idle timeout expired: 300001/300000 ms

After the disconnection, or when the previous console session attempts to
reconnect, the CPVM log reports:

External authenticator failed

The browser WebSocket initially completes its handshake with HTTP status 101.

During the failure:

• The guest VM remains Running.
• The Console Proxy VM remains Running.
• The Console Proxy agent remains Up.
• The CPVM public interface and TCP port 8080 remain reachable.
• The KVM VNC listener remains active.
• Reopening the console from the CloudStack UI immediately works with a newly
  generated token.

The observed traffic path is:

Browser → CPVM public interface TCP 8080 → CPVM private interface →
KVM VNC endpoint → guest VM

versions

• Apache CloudStack Management Server: 4.22.1.1
• cloudstack-common: 4.22.1.1
• cloudstack-agent: 4.22.1.1
• System VM template/version: 4.22.1.1
• Management Server OS: Ubuntu 24.04 LTS
• KVM Host OS: Ubuntu 24.04 LTS
• Hypervisor: KVM/libvirt
• Host and System VM architecture: aarch64
• Browser: Google Chrome
• Zone network type: Advanced
• Console Proxy VM state: Running
• Console Proxy agent state: Up
• novnc.console.default: true
• consoleproxy.session.timeout: 300000

No reverse proxy or external load balancer is placed between the browser and
the Console Proxy VM.

The steps to reproduce the bug

1. Deploy and start a guest VM on an aarch64 KVM host.
2. Confirm that the guest VM and Console Proxy VM are Running and their agents
   are Up.
3. Open the guest console from the CloudStack UI using the default noVNC
   console.
4. Confirm that the browser WebSocket handshake returns HTTP status 101.
5. Continuously interact with the console using keyboard input and observe
   ongoing screen updates.
6. Observe that the console disconnects after approximately 2–5 minutes,
   despite the active session.
7. Confirm that the guest VM remains Running.
8. Confirm that the CPVM public interface and TCP port 8080 remain reachable.
9. Confirm that the KVM VNC listener remains active.
10. Observe Idle timeout expired: 300001/300000 ms in the CPVM log.
11. Observe External authenticator failed when the previous session attempts
    to reconnect, or observe the browser message
    Failed to connect to server / access token has expired.
12. Close the old console window and open the console again from the
    CloudStack UI.
13. Confirm that the newly generated token restores console access
    temporarily.

What to do about it?

Expected behavior:

An actively used noVNC console should remain connected while the guest VM,
Console Proxy VM, network path, and KVM VNC backend remain healthy.

Active keyboard input, WebSocket traffic, and VNC framebuffer updates should
prevent the connection from being treated as idle.

If consoleproxy.session.timeout is intended to apply to noVNC sessions, the
configured value should be handled consistently by the Jetty WebSocket idle
timeout and the Console Proxy viewer garbage-collection logic.

Please confirm:

1. Whether this behavior is addressed by PR Honour consoleproxy.session.timeout for noVNC sessions (fixes #12810) #13002.
2. Whether that fix is included in any released 4.22 package.
3. Whether a backport to the maintained 4.22 branch is planned.
4. Whether there is a supported workaround for CloudStack 4.22.1.1.
5. Whether consoleproxy.session.timeout should control both the Jetty
   WebSocket idle timeout and Console Proxy viewer garbage collection.
6. Whether applying a changed timeout requires restarting the Management
   Server, restarting the Console Proxy VM, or recreating the Console Proxy VM.

Security note

Console URLs, authentication tokens, API credentials, passwords, SSH keys,
MAC addresses, UUIDs, and sensitive infrastructure identifiers have been
removed from this report.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[montagnahuanghsiao]

Additional timeout configuration evidence

The current CloudStack global setting is:

consoleproxy.session.timeout = 28800000

This corresponds to 8 hours:

28800000 ms = 8 hours

However, during the reproduced noVNC console failure, the Console Proxy VM
logged:

Idle timeout expired: 300001/300000 ms

The effective Jetty WebSocket idle timeout was therefore still 300000 ms
(5 minutes), rather than the configured 28800000 ms (8 hours).

The affected Console Proxy implementation also appears to contain a separate
hard-coded value:

ConsoleProxy.VIEWER_LINGER_SECONDS = 180

This creates three different timeout values that may affect the same console
session lifecycle:

• Configured consoleproxy.session.timeout: 28800000 ms (8 hours)
• Observed Jetty WebSocket idle timeout: 300000 ms (5 minutes)
• ConsoleProxy.VIEWER_LINGER_SECONDS: 180 seconds (3 minutes)

This may explain why the console can disconnect after approximately
2–5 minutes even though the configured session timeout is 8 hours.

The console was actively being used when the disconnection occurred. The guest
VM remained Running, the Console Proxy VM remained Running with Agent State Up,
and reopening the console from the CloudStack UI immediately worked with a new
access token.

Could you please confirm:

1. Whether the Jetty WebSocket idle timeout is expected to use
   consoleproxy.session.timeout.
2. Whether ConsoleProxy.VIEWER_LINGER_SECONDS is expected to remain
   hard-coded at 180 seconds.
3. Whether PR Honour consoleproxy.session.timeout for noVNC sessions (fixes #12810) #13002 makes both the WebSocket timeout and viewer
   garbage-collection timeout honor consoleproxy.session.timeout.
4. Whether the fix is included in any released or planned 4.22 package.
5. Whether changing consoleproxy.session.timeout requires restarting the
   Management Server, restarting the Console Proxy VM, or recreating the
   Console Proxy VM.