Disable TX ip checksum offload in VMware ESXi 5.5.0 Update 2

[vincentmli]

Just file an issue to track this, while running F-Stack Nginx in VM in VMware ESXi 5.5.0 Update 2, the TX ip checksum offload is enabled by default, but the the ip checksum is not calculated in the hardware, result in ip checksum value 0

# tcpdump -nn  -v -r /tmp/cpu0_0.pcap  host 10.1.72.28 and tcp | head -4

SYN from client 10.1.72.28  to  F-Stack Nginx server 10.1.72.168

23:06:37.104023 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.72.28.1025 > 10.1.72.168.80: Flags [S], cksum 0xd9ce (correct), seq 1012484, win 14600, options [mss 1460,sackOK,TS val 2692380804 ecr 0,nop,wscale 7], length 0

SYN+ACK from F-Stack Nginx server 10.1.72.168 to client, the ip checksum is 0, and dropped by client.

23:06:37.104023 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60, bad cksum 0 (->95f6)!)
    10.1.72.168.80 > 10.1.72.28.1025: Flags [S.], cksum 0xa4f4 (incorrect -> 0x656f), seq 2977315146, ack 1012485, win 8192, options [mss 1460,nop,wscale 9,sackOK,TS val 237605994 ecr 2692380804], length 0

After digging through the code and found tx_csum_offoad_skip config in [dpdk], and #317

            if (ff_global_cfg.dpdk.tx_csum_offoad_skip == 0) {
                if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_IPV4_CKSUM)) {
                    rte_log(RTE_LOG_INFO, RTE_LOGTYPE_USER1, "TX ip checksum offload supported\n");
                    port_conf.txmode.offloads |= DEV_TX_OFFLOAD_IPV4_CKSUM;
                    pconf->hw_features.tx_csum_ip = 1;
                }

                if ((dev_info.tx_offload_capa & DEV_TX_OFFLOAD_UDP_CKSUM) &&
                    (dev_info.tx_offload_capa & DEV_TX_OFFLOAD_TCP_CKSUM)) {
                    rte_log(RTE_LOG_INFO, RTE_LOGTYPE_USER1, "TX TCP&UDP checksum offload supported\n");
                    port_conf.txmode.offloads |= DEV_TX_OFFLOAD_UDP_CKSUM | DEV_TX_OFFLOAD_TCP_CKSUM;
                    pconf->hw_features.tx_csum_l4 = 1;
                }

disable ip checksum offload resolves the tcp connection issue, this also affect icmp ping since it is ip packet.

tx_csum_offoad_skip=1

[vincentmli]

I wonder if we can introduce another configuration option to disable TX ip checksum offload only, and keep tcp/udp checksum offload enabled, I did a test with that and it works in VMware ESXi 5.5.0 Update 2 environment.

the SYN+ACK from F-Stack Nginx, no more "bad cksum 0" in the ip header, but keep tcp checksum offload enabled

# tcpdump -n -v -r /tmp/cpu0_0.pcap host 10.1.72.28

15:53:16.597155 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    10.1.72.168.80 > 10.1.72.28.1025: Flags [S.], cksum 0xa4f4 (incorrect -> 0x2b07), seq 1851342229, ack 1012485, win 8192, options [mss 1460,nop,wscale 9,sackOK,TS val 824592216 ecr 2752780346], length 0

[jfb8856606]

Yes, you can create a pull requeset to F-Stack.

[jfb8856606]

The workaround for this issue is to set tx_csum_offoad_skip=1 in the [dpdk] section of config.ini, which disables all TX checksum offloading (IP + TCP/UDP). This resolves the problem in VMware and other virtualized environments where the NIC advertises TX checksum offload capability but doesn't actually calculate the checksum.

The finer-grained control suggested by @vincentmli (disable only IP checksum offload while keeping TCP/UDP checksum offload enabled) has not been implemented yet. This would be a welcome contribution — the relevant code is in lib/ff_dpdk_if.c around the tx_csum_offoad_skip check, and the hw_features.tx_csum_ip / hw_features.tx_csum_l4 fields already exist separately, so the infrastructure for per-layer control is partially in place.

See also #317 for the same root cause.

Keeping this issue open as the finer-grained TX checksum offload control has not been implemented.