Debugging ARMv8 system clock issue

I noticed one issue with my board - the iperf server and client shows different performance data. Since the client is Windows PC and that has the correct clock, the clock on my embedded linux must be wrong. Here is how I debugged the issue

First figure out the source that is being used as clock

# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
arch_sys_counter
# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
arch_sys_counter
# cat /proc/interrupts
           CPU0       CPU1
  1:          0          0     GICv3  29 Edge      arch_timer
  2:       1117         93     GICv3  30 Edge      arch_timer
  5:       4558          0     GICv3  44 Level     serial_tx
  6:        515          0     GICv3  45 Level     serial_rx
  8:          0          0     GICv3 192 Level     xhci-hcd:usb1
  9:          0          0     GICv3 194 Level     a0020000.sata
 10:          0          0     GICv3 196 Level     mvri-irq
IPI0:       278        554       Rescheduling interrupts
IPI1:         4          4       Function call interrupts
IPI2:         0          0       CPU stop interrupts
IPI3:         0          0       Timer broadcast interrupts
IPI4:         0          0       IRQ work interrupts

It is the ARMv-8 arch counter, which is in the IP from ARM and every ARMv8 system should have one. Let's check if there is anything else related to clock from kernel message

# dmesg | grep clock
[    0.000000] clocksource arch_sys_counter: mask: 0xffffffffffffff max_cycles:
0x49cd42e20, max_idle_ns: 440795202120 ns
[    0.000003] sched_clock: 56 bits at 20MHz, resolution 50ns, wraps every 43980
46511100ns
[    0.005003] Registered arch_counter_get_cntvct+0x0/0x10 as sched_clock source

[    0.306657] clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max
_idle_ns: 95563022313750000 ns
[    0.510202] Switched to clocksource arch_sys_counter

It said 56bits counter at 20MHz. Where was this value come from? Let's dig a little bit more. Grep "arch_sys_counter" and follow the calling stack finally I found the following code detect the clock frequency

static void
arch_timer_detect_rate(void __iomem *cntbase, struct device_node *np)
{
        /* Who has more than one independent system counter? */
        if (arch_timer_rate)
                return;

        /*
         * Try to determine the frequency from the device tree or CNTFRQ,
         * if ACPI is enabled, get the frequency from CNTFRQ ONLY.
         */
        if (!acpi_disabled ||
            of_property_read_u32(np, "clock-frequency", &arch_timer_rate)) {
                if (cntbase)
                        arch_timer_rate = readl_relaxed(cntbase + CNTFRQ);
                else
                        arch_timer_rate = arch_timer_get_cntfrq();
        }

        printk(KERN_INFO "arch_timer_rate is %08x , cntbase is %p. \n", 
            arch_timer_rate, cntbase);

        /* Check the timer frequency. */
        if (arch_timer_rate == 0)
                pr_warn("Architected timer frequency not available\n");
}

The code above first try to detect the clock frequency from device tree (in which I did not setup the timer entry), then try to get it from arch_timer_get_cntfrq().

vim arch/arm64/include/asm/arch_timer.h
static inline u32 arch_timer_get_cntfrq(void)
{
        u32 val;
        asm volatile("mrs %0,   cntfrq_el0" : "=r" (val));
        return val;
}

The clock frequency is in co-processor cntfrq_el0. Let's grep if the value of arch_timer_rate was printed out
# dmesg | grep arch_timer_rate
[    0.000000] arch_timer_rate is 01312d00 , cntbase is           (null).

Note 0x01312d00 is 20M. Is this value a default value on reset or was it preset by a piece of code? I grep "cntfrq_el0" in my kernel source codes and found nothing, and I tried to grep in u-boot's code to find this

 vim u-boot/arch/arm/cpu/armv8/nap/psci.S
 .global _armadalp_cpu_entry
_armadalp_cpu_entry:

        bl      enable_affinity

        isb

        /*
         * Could be EL3/EL2/EL1, Initial State:
         * Little Endian, MMU Disabled, i/dCache Disabled
         */
        adr     x0, vectors
        switch_el x1, 3f, 2f, 1f
3:      msr     vbar_el3, x0
        mrs     x0, scr_el3
        orr     x0, x0, #0xf                    /* SCR_EL3.NS|IRQ|FIQ|EA */
        msr     scr_el3, x0
        msr     cptr_el3, xzr                   /* Enable FP/SIMD */
        ldr     x0, =COUNTER_FREQUENCY
        msr     cntfrq_el0, x0                  /* Initialize CNTFRQ */
        b       0f
2:      msr     vbar_el2, x0
        mov     x0, #0x33ff
        msr     cptr_el2, x0                    /* Enable FP/SIMD */
        b       0f
1:      msr     vbar_el1, x0
        mov     x0, #3 << 20
        msr     cpacr_el1, x0                   /* Enable FP/SIMD */
0:

Register "cntfrq_el0" was initialized as COUNTER_FREQUENCY which was defined as 20M in a header file.  This is the value we used on another chip and I ported the u-boot based on that chip but I never got chance to modify this value. I talked with chip designer and figured out it shoud be  12.5M on our system.  Issue fixed by update the constant marco COUNTER_FREQUENCY.