Linux kernel CRC checksum error

To remove the possible performance impact caused by NFS, I tried to add several test programs into my rootfs. First I added a 3M bytes file, everything is fine, then I added a lot of files with 18Mbytes in total, then suddenly my kernel can not be loaded.

U-Boot code: 00000000 -> 00047568  BSS: -> 00065340
DRAM:  Monitor len: 00065340
Ram size: 40000000
Ram top: 40000000
TLB table from 3fff0000 to 40000000
Reserving 404k for U-Boot at: 3ff8a000
Reserving 5184k for malloc() at: 3fa7a000
Reserving 176 Bytes for Board Info at: 3fa79f50
Reserving 408 Bytes for Global Data at: 3fa79db8
Reserving 11264 Bytes for FDT at: 3fa771b8
DRAM:  1 GiB
New Stack Pointer is: 3fa771a0
Relocation Offset is: 3ff8a000
Relocating to 3ff8a000, new gd at 3fa79db8, sp at 3fa771a0
Initializing board
Board: DB-88F3720-DDR3-Modular
       CPU     @ 600    [MHz]
       DDR     @ 600    [MHz]
Now running in RAM - U-Boot at: 3ff8a000
U-Boot DT blob at : 000000003fa771b8

Hit any key to stop autoboot:  0
## Booting kernel from Legacy Image at 0017ffc0 ...
   Image Name:   linux-4.1.6
   Image Type:   AArch64 Linux Kernel Image (uncompressed)
   Data Size:    15304304 Bytes = 14.6 MiB
   Load Address: 00180000
   Entry Point:  00180000
   Verifying Checksum ... Bad Data CRC
ERROR: can't get kernel image!

Weird thing is, if I load the kernel after u-boot already reach the prompt, everything is good. It looks U-boot write something into the kernel area during the init phase. How come? My first thought is to compare. U-boot has the command "cmp" to compare 2 chunk of data, what I need to do is to load an extra copy of my kernel into memory

LOAD /BINARY /ADDRESS=0x1017FFC0 OF "C:\incoming\bin\kernel_image"

Note my kernel was loaded to 0x0017FFC0 together with u-boot. Now do the compare with u-boot

U-boot>> cmp.b 0x0017ffc0 0x1017ffc0 15304308
byte at 0x0000000000fef4e0 (0x30) != byte at 0x0000000010fef4e0 (0x0)
Total of 15136032 byte(s) were the same
U-boot>> cmp.l 0xfef4e0 0x10fef4e0 0x10000
word at 0x0000000000fef4e0 (0xfef530) != word at 0x0000000010fef4e0 (0x100)
Total of 0 word(s) were the same

Let's dump the memory

U-boot>> md.l 0xfef4e0 0x100
00fef4e0: 00fef530 00000000 00020b44 00000000    0.......D.......
00fef4f0: ffffffff 00000000 00fef720 00000000    ........ .......
00fef500: 00000000 00000000 ffffffff 00000000    ................
00fef510: 0000000a 00000000 00000000 00000000    ................
00fef520: ffffffff 00000000 00000001 00000000    ................
00fef530: 00fef5e0 00000000 00021460 00000000    ........`.......
00fef540: 00fefc00 00000000 0003b527 00000000    ........'.......
00fef550: 00fef720 00000000 000313c8 00000000     ...............
00fef560: ffffffd0 00000000 00fef650 00000000    ........P.......
00fef570: 10004001 00004101 00fefc00 00000000    .@...A..........
00fef580: 00040000 000a4204 000313c8 00000000    .....B..........
00fef590: 00fef640 00000000 00021460 00000000    @.......`.......
00fef5a0: 00fefc60 00000000 0003494b 00000000    `.......KI......
00fef5b0: 00fef780 00000000 000313c8 00000000    ................
00fef5c0: ffffffe0 00000000 00fef6b0 00000000    ................
00fef5d0: 10004001 00004101 00fefc60 00000000    .@...A..`.......
00fef5e0: 00040000 000a4204 00021034 00000000    .....B..4.......
00fef5f0: 00fefc30 00000000 37316130 33666137    0.......0a177af3
00fef600: 00fef750 00000000 000313c8 00000000    P...............
00fef610: 00fef6b0 00000000 00021720 00000000    ........ .......
00fef620: 00fef750 00000000 0003b52e 00000000    P...............
00fef630: 40000000 00000000 00034776 00000000    ...@....vG......
00fef640: 00fef6e0 00000000 00021720 00000000    ........ .......
00fef650: 00fef780 00000000 0003491b 00000000    .........I......
00fef660: 00000000 00000000 40076d90 00000000    .........m.@....
00fef670: 00000000 00000000 00000000 00000000    ................
00fef680: 00000000 00800020 00000400 08010000    .... ...........
00fef690: 00040000 000a4204 00000d80 02008480    .....B..........
00fef6a0: 00fef6c0 00000000 000101e8 00000000    ................
00fef6b0: 00fef6c0 00000000 0001684c 00000000    ........Lh......
00fef6c0: 00fef6e0 00000000 0001694c 00000000    ........Li......
00fef6d0: 0000000a 00000000 000101e8 00000000    ................
00fef6e0: 00fef700 00000000 000168b4 00000000    .........h......
00fef6f0: 00fef7bb 00000000 0001689c 00000000    .........h......
00fef700: 00fef720 00000000 000101e8 00000000     ...............
00fef710: 0000003b 00000000 0003e198 00000000    ;...............
00fef720: 00fefc60 00000000 00004864 00000000    `.......dH......
00fef730: 0003e2a8 00000000 00fefc30 00000000    ........0.......
00fef740: 00fefc60 00000000 00fefc60 00000000    `.......`.......
00fef750: 00fefc20 00000000 ffffffc8 ffffff80     ...............
00fef760: 00fefc60 00000000 00fefc60 00000000    `.......`.......
00fef770: 00fefc20 00000000 ffffffc8 ffffff80     ...............
00fef780: 6f6c6552 69746163 7420676e 6633206f    Relocating to 3f
00fef790: 30613866 202c3030 2077656e 61206467    f8a000, new gd a
00fef7a0: 66332074 64393761 202c3862 61207073    t 3fa79db8, sp a
00fef7b0: 66332074 31373761 000a3061 36313032    t 3fa771a0..2016
00fef7c0: 2e31545f 6e652e30 72645f67 765f706f    _T1.0.eng_drop_v
00fef7d0: 000a0a32 00000000 00254e30 ffffffc0    2.......0N%.....
00fef7e0: 002554c8 ffffffc0 00000002 00000000    .T%.............
00fef7f0: 00000000 00000000 00000000 00000000    ................
00fef800: 00000000 00000000 00000000 00000000    ................
00fef810: 00000000 00000000 00fef878 ffffffc0    ........x.......
00fef820: 00fef858 ffffffc0 00fef8d8 ffffffc0    X...............
00fef830: 00fef8b8 ffffffc0 00fef898 ffffffc0    ................
00fef840: 00fef918 ffffffc0 00fef8f8 ffffffc0    ................
00fef850: 00000000 00000000 00699768 ffffffc0    ........h.i.....
00fef860: 000001a4 00000000 002550d0 ffffffc0    .........P%.....
00fef870: 00254ea0 ffffffc0 00699778 ffffffc0    .N%.....x.i.....
00fef880: 000001a4 00000000 00255100 ffffffc0    .........Q%.....
00fef890: 00255618 ffffffc0 00698958 ffffffc0    .V%.....X.i.....
00fef8a0: 00000124 00000000 00255040 ffffffc0    $.......@P%.....
00fef8b0: 00000000 00000000 00699780 ffffffc0    ..........i.....
00fef8c0: 00000124 00000000 00255070 ffffffc0    $.......pP%.....
00fef8d0: 00000000 00000000 006989b0 ffffffc0    ..........i.....

Look at the string "Relocating to"----it looks like this area is being used as a stack area by u-boot. What was the stack pointer when u-boot print "Relocating to"?  It was in function "setup_reloc()" but my debugger do not like the idea to set a break point on this function so I have to add an endless while loop



The stack pointer is 0xFEFC60, which is close to the mis-compare area. Of course it won't work because this stack conflict with my kernel image.  When and where is this stack pointer set? Set a break point on _main and I got



The initial stack pointer was defined by CONFIG_SYS_INIT_SP_ADDR. Refresh the definition and now the kernel can be loaded with large rootfs.

Debugging ARMv8 system clock issue

I noticed one issue with my board - the iperf server and client shows different performance data. Since the client is Windows PC and that has the correct clock, the clock on my embedded linux must be wrong. Here is how I debugged the issue

First figure out the source that is being used as clock

# cat /sys/devices/system/clocksource/clocksource0/available_clocksource
arch_sys_counter
# cat /sys/devices/system/clocksource/clocksource0/current_clocksource
arch_sys_counter
# cat /proc/interrupts
           CPU0       CPU1
  1:          0          0     GICv3  29 Edge      arch_timer
  2:       1117         93     GICv3  30 Edge      arch_timer
  5:       4558          0     GICv3  44 Level     serial_tx
  6:        515          0     GICv3  45 Level     serial_rx
  8:          0          0     GICv3 192 Level     xhci-hcd:usb1
  9:          0          0     GICv3 194 Level     a0020000.sata
 10:          0          0     GICv3 196 Level     mvri-irq
IPI0:       278        554       Rescheduling interrupts
IPI1:         4          4       Function call interrupts
IPI2:         0          0       CPU stop interrupts
IPI3:         0          0       Timer broadcast interrupts
IPI4:         0          0       IRQ work interrupts

It is the ARMv-8 arch counter, which is in the IP from ARM and every ARMv8 system should have one. Let's check if there is anything else related to clock from kernel message

# dmesg | grep clock
[    0.000000] clocksource arch_sys_counter: mask: 0xffffffffffffff max_cycles:
0x49cd42e20, max_idle_ns: 440795202120 ns
[    0.000003] sched_clock: 56 bits at 20MHz, resolution 50ns, wraps every 43980
46511100ns
[    0.005003] Registered arch_counter_get_cntvct+0x0/0x10 as sched_clock source

[    0.306657] clocksource jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max
_idle_ns: 95563022313750000 ns
[    0.510202] Switched to clocksource arch_sys_counter

It said 56bits counter at 20MHz. Where was this value come from? Let's dig a little bit more. Grep "arch_sys_counter" and follow the calling stack finally I found the following code detect the clock frequency

static void
arch_timer_detect_rate(void __iomem *cntbase, struct device_node *np)
{
        /* Who has more than one independent system counter? */
        if (arch_timer_rate)
                return;

        /*
         * Try to determine the frequency from the device tree or CNTFRQ,
         * if ACPI is enabled, get the frequency from CNTFRQ ONLY.
         */
        if (!acpi_disabled ||
            of_property_read_u32(np, "clock-frequency", &arch_timer_rate)) {
                if (cntbase)
                        arch_timer_rate = readl_relaxed(cntbase + CNTFRQ);
                else
                        arch_timer_rate = arch_timer_get_cntfrq();
        }

        printk(KERN_INFO "arch_timer_rate is %08x , cntbase is %p. \n", 
            arch_timer_rate, cntbase);

        /* Check the timer frequency. */
        if (arch_timer_rate == 0)
                pr_warn("Architected timer frequency not available\n");
}

The code above first try to detect the clock frequency from device tree (in which I did not setup the timer entry), then try to get it from arch_timer_get_cntfrq().

vim arch/arm64/include/asm/arch_timer.h
static inline u32 arch_timer_get_cntfrq(void)
{
        u32 val;
        asm volatile("mrs %0,   cntfrq_el0" : "=r" (val));
        return val;
}

The clock frequency is in co-processor cntfrq_el0. Let's grep if the value of arch_timer_rate was printed out
# dmesg | grep arch_timer_rate
[    0.000000] arch_timer_rate is 01312d00 , cntbase is           (null).

Note 0x01312d00 is 20M. Is this value a default value on reset or was it preset by a piece of code? I grep "cntfrq_el0" in my kernel source codes and found nothing, and I tried to grep in u-boot's code to find this

 vim u-boot/arch/arm/cpu/armv8/nap/psci.S
 .global _armadalp_cpu_entry
_armadalp_cpu_entry:

        bl      enable_affinity

        isb

        /*
         * Could be EL3/EL2/EL1, Initial State:
         * Little Endian, MMU Disabled, i/dCache Disabled
         */
        adr     x0, vectors
        switch_el x1, 3f, 2f, 1f
3:      msr     vbar_el3, x0
        mrs     x0, scr_el3
        orr     x0, x0, #0xf                    /* SCR_EL3.NS|IRQ|FIQ|EA */
        msr     scr_el3, x0
        msr     cptr_el3, xzr                   /* Enable FP/SIMD */
        ldr     x0, =COUNTER_FREQUENCY
        msr     cntfrq_el0, x0                  /* Initialize CNTFRQ */
        b       0f
2:      msr     vbar_el2, x0
        mov     x0, #0x33ff
        msr     cptr_el2, x0                    /* Enable FP/SIMD */
        b       0f
1:      msr     vbar_el1, x0
        mov     x0, #3 << 20
        msr     cpacr_el1, x0                   /* Enable FP/SIMD */
0:

Register "cntfrq_el0" was initialized as COUNTER_FREQUENCY which was defined as 20M in a header file.  This is the value we used on another chip and I ported the u-boot based on that chip but I never got chance to modify this value. I talked with chip designer and figured out it shoud be  12.5M on our system.  Issue fixed by update the constant marco COUNTER_FREQUENCY.

nfs mount RPC service issue

Ported a 4.1 kernel to ARMv8/CA72 board.  I always failed with the following command

# mount 10.85.151.108:/srv /mnt/nfs
[  241.430956] svc: failed to register lockdv1 RPC service (errno 111).
mount: mounting 10.85.151.108:/srv on /mnt/nfs failed: Connection refused

Finally I have to use this
# mount -o port=2049,nolock,proto=tcp 10.85.151.108:/srv /mnt/nfs

Still trying to figure out why udp is not working.

mutex in armv8

Finally I am working on armv8 soc now. At the very beginning I need a mutex, which is essential for multiple core processors. The old strex/lderx on longer available, now we have ldxr/stxr instead

http://infocenter.arm.com/help/index.jsp?topic=/com.arm.doc.dht0008a/ch01s03s02.html


get_lock PROC
mov w1, #0x1
try_loop
ldaxr w2, [x0]
cbnz w2, try_loop
stxr w2, w1, [x0]
cbnz w2, try_loop
ret
ENDP


free_lock PROC
mov w1, #0x0
stlr w1, [x0]
ret
ENDP

try_lock PROC
mov w1, #0x1
ldaxr w2, [x0]
cbnz w2, exit
stxr w2, w1, [x0]
exit
mov w0, w2
ret
ENDP