path: root/md/writeup/raspberry5_baremetal_gic_and_timers.md

title: Raspberry 5 baremetal GIC and timer's
keywords:raspi5,asm,c,kernel,arm64,gic,timer,irq,armstub

# Raspberry 5 bare metal GIC and timer's

# Intro

Bringing up timers for Raspberry PI 5. There is 2 options ARM ones and BCM one. 
Then idea is to create irq handlers and handle timer events. Change from previouse
versions of raspberry PI is that BCM timer now using GIC rather then BCM IRQ's.

## ARM generic timer

ARM support generic timer's that come out of the box for ARM SoC's

Linux kerne defines armv8 timers as
```dts
timer {
		compatible = "arm,armv8-timer";
		interrupts = <GIC_PPI 13 (GIC_CPU_MASK_SIMPLE(4) |
					  IRQ_TYPE_LEVEL_LOW)>,
			     <GIC_PPI 14 (GIC_CPU_MASK_SIMPLE(4) |
					  IRQ_TYPE_LEVEL_LOW)>,
			     <GIC_PPI 11 (GIC_CPU_MASK_SIMPLE(4) |
					  IRQ_TYPE_LEVEL_LOW)>,
			     <GIC_PPI 10 (GIC_CPU_MASK_SIMPLE(4) |
					  IRQ_TYPE_LEVEL_LOW)>,
			     <GIC_PPI 12 (GIC_CPU_MASK_SIMPLE(4) |
					  IRQ_TYPE_LEVEL_LOW)>;
	};
```

There are one of possibilities to run as timers.


## BCM2835 timer

Timer allows to set value and when the value matches the counter interrupt is rised.
Can be used for system tick's or sheduling events. Its seem's esiers peripheral to
start with. 

### Interrupts

Linux kernel defines interrupts in device-tree as 

```dts
system_timer: timer@7c003000 {
	compatible = "brcm,bcm2835-system-timer";
	reg = <0x7c003000 0x1000>;
	interrupts = <GIC_SPI 64 IRQ_TYPE_LEVEL_HIGH>,
		     <GIC_SPI 65 IRQ_TYPE_LEVEL_HIGH>,
		     <GIC_SPI 66 IRQ_TYPE_LEVEL_HIGH>,
		     <GIC_SPI 67 IRQ_TYPE_LEVEL_HIGH>;
	clock-frequency = <1000000>;
};
```

| GIC_SPI | Actual number | Note |
|---|---|---|
| 64 | 96 | Timer C0 |
| 65 | 97 | Timer C1 |
| 66 | 98 | Timer C2 |
| 67 | 99 | Timer C3 |

### System Timer Registers 

Address where are mapped was __0x107c003000__

| Offset | Register | Name | Description |
|---|---|---|---|
| 0x0  | CS  | System Timer Control/Status         |  |
| 0x4  | CLO | System Timer Counter Lower 32 bits  | |
| 0x8  | CHI | System Timer Counter Higher 32 bits | |
| 0xc  | C0  | System Timer Compare 0              | |
| 0x10 | C1  | System Timer Compare 1              | |
| 0x14 | C2  | System Timer Compare 2              | |
| 0x18 | C3  | System Timer Compare 3              | |

### Usage of the driver

Clear Control registers, is just writing bit's that need to be cleared
```c
#define bcm2835_read32(offset) (*(volatile uint32_t *)(TM_BCM2835_REG_OFFSET+offset))
#define bcm2835_write32(offset,val) (*(volatile uint32_t *)(TM_BCM2835_REG_OFFSET+offset)) = val


void timer_clr_stat() {
    bcm2835_write32(BCM2835_CS,0xF);
}
```

To set timer event, need to read counter register and add timer interval to that value 
```c
void timer_start_c0() {
    uint32_t t_lo = bcm2835_read32(BCM2835_CLO);
    t_lo += TIMER_INTERVAL;
    bcm2835_write32(BCM2835_C0, t_lo);
}
```

## GIC

GIC - Generic Interrupt Controller 

Comparing to other ones that I have worked with this is more complex and support
different modes. All ARM SoC's will have it and in one or other way will need to deal
with it, so its interesting one to check how it works.

GIC consist from few parts depend on version. RP5 uses GIC-400

### GIC Register list

| Register | Offset | Note |
|---|---|---|
| GICC_CTRL | 0x0  | Enable disable IRQ types, configure IRQ group behaviour |
| GICC_PMR  | 0x4  | IRQ priority filter |
| GICC_IAR  | 0xC  | Read current IRQ to handle |
| GICC_EOIR | 0x10 | Notify that IRQ is handled |
| GICC_APR0 | 0xD0 | State management  |
| GICC_APR1 | 0xD4 | State management |
| GICC_APR2 | 0xD8 | State management |
| GICC_APR3 | 0xDC | State management |

### GICD Register list

| Register       | Offset | Note |
|---|---|---|
|GICD_CTRL       | 0x0   | Enable/Disable forward IRQ's |
|GICD_IDR        | 0x8   | Identification register |
|GICD_IGROUPR    | 0x080 | Set interrupt group |
|GICD_ISENABLER  | 0x100 | Set enable IRQ|
|GICD_ICENABLER  | 0x180 | Clear enable IRQ |
|GICD_ISPENDR    | 0x200 | Set pending interrupt |
|GICD_ICPENDR    | 0x280 | Clear pending interrupt |
|GICD_ISACTIVER  | 0x300 | Set active interrupt |
|GICD_ICACTIVER  | 0x380 | Clear active interrupt |
|GICD_IPRIORITYR | 0x400 | Set IRQ priority |
|GICD_ITARGETSR  | 0x800 | Set target CPU |
|GICD_ICFGR      | 0xC00 | Interrupt config |

### Initializing GIC

### Setting up GIC for use
Setting up GIC. Switch of Controller and Distributor, clear all pending and active statuses.
Set all priority filters, and enable all groups. And then enable Controller and Distributor.

```c
iowrite32(GICD_CTRL, 0);
iowrite32(GICC_CTRL, 0);

for (i = 0; i < 16; i++) {
	iowrite32(GICD_ICENABLER(i), 0xffffffff);
	iowrite32(GICD_ICPENDR(i),   0xffffffff);
	iowrite32(GICD_ICACTIVER(i), 0xffffffff);
}
for (i = 0; i < 16; i++)
	iowrite32(GICD_IGROUPR(i), 0xffffffff);
for (i = 0; i < 128; i++) {
	iowrite32(GICD_IPRIORITYR(i), 0xa0a0a0a0); 
	iowrite32(GICD_ITARGETSR(i),  0x01010101); 
}
for (i = 0; i < 32; i++)
	iowrite32(GICD_ICFGR(i), 0x55);
iowrite32(GICD_CTRL, 0x1);
iowrite32(GICC_PMR, 0xff);
iowrite32(GICC_CTRL, 0x3);
```

### Draining interrupt queue before use. 

There was some time spent to figure out why IAR report's spuriouse interrupts
so code added that drains all interrupts and restart state. Issue with enabling
all IRQ's there was 3 extra interrupts that for some reason where raised. Probably
something GPU related. And GIC state after few kernel loads in gdb where messed up.


drain all pending interrupts
```c
uint32_t val;
while (((val = ioread32(GICC_IAR)) & 0x3ff) < 1020)
	iowrite32(GICC_EOIR, val);
```

Restart state machine

```c
iowrite32(GICC_APR0, 0);
iowrite32(GICC_APR1, 0);
iowrite32(GICC_APR2, 0);
iowrite32(GICC_APR3, 0);
```

### Enable interrupt handling

```c
iowrite32(GICD_ISENABLER(irqn/32), 1 << (irqn % 32));
```

### GDB notes 
Checking status of GIC_IAR register in gdb

```
p/x *0x107fffa00c
```

if value is __0x3ff__ its spurious interrupt. I seen this state after loading kernel few times
and couldn't recover from that option was to restart the board or use GICC_APR register's to clear
the state. 

## Execution levels

Pi 5 launches from EL2 level and that why not possible to configure interrupts. So we
need to modify armstub to be able to do that. 

To check execution level do 
```c
static inline uint32_t current_el(void) {
    uint64_t el;
    __asm__ volatile("mrs %0, CurrentEL" : "=r"(el));
    return (uint32_t)((el >> 2) & 0x3);
}
``` 

On my machine it got reported as EL2 without any stub configuration
there is possible to configure arm stub that will be executed before loading kernel

```
armstub=armstubv8.bin
```

base example can be found at https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S  
I tried to modify and run it, Execution level was reported as EL3, seems it failed and need more
investigation then just bare replacment of addresses

## IRQ handling

To handle IRQ's need to setup vector table and passed to VBAR register, and irq handling 
function


### Vector tables

```arm
.align 11
sync_exception_handler:
    bl kernel_panic
    eret

.macro ventry label
    .align 7
    b \label
.endm

dummy_vector:
    b 0x200

.align 11
vector_table:
    ventry sync_exception_handler
    ventry dummy_vector 
    ventry dummy_vector 
    ventry dummy_vector 
    
    //EL1h
    ventry dummy_vector 
    ventry handle_irq
    ventry dummy_vector 
    ventry dummy_vector 
```

Vector table address can be passed as 
for EL2

```arm
ldr x1, =vector_table
msr VBAR_EL2, x1
isb
```
for EL3
```
ldr x1, =vector_table
msr VBAR_EL3, x1
isb
```

It took some time to figure out why EL3 part didn't worked, only after
checking execution level right approach was choosen(it was EL2). Also when I did
attempted to run stub I got EL3 rather then EL2.

### IRQ handler

IRQ handler function will be called when generic IRQ is triggered, so it will need
to handle each of the IRQ's is some way

```c
void irq_handler() {
    uint32_t val = ioread32(GICC_IAR);
    uint32_t irq = val & 0x3ff;

    if (irq >= 1020) {
        return;
    }

    pl011_puts("Got IRQ ");
    pl011_putu32(irq);
    pl011_puts("\r\n");
    bcm2835_write32(BCM2835_CS, 0xf);  // clear all four match flags
    timer_start_c1();

    iowrite32(GICC_EOIR, val);
}
```

## Sumup

There is many parts that's need's to be combined to make IRQ's to work

1. Setting up the vector table
2. Write vector table address
3. Enable the interrupts in GIC
4. Enable global interrupts
5. Set timer
6. Handle interrupt

Some parts needs to be cleaned up

## ARMv8 Stub's

Raspberry Pi provide option to boot custom armstub's

## Source

## Links

https://www.raspberrypi.com/documentation/computers/config_txt.html  
https://forums.raspberrypi.com/viewtopic.php?t=371974  
https://docs.amd.com/r/en-US/ug585-zynq-7000-SoC-TRM/Shared-Peripheral-Interrupts-SPI?tocId=jlBXQc~lyPkRtaPCJuxp2Q  
https://developer.arm.com/documentation/ihi0048/b/Programmers--Model/CPU-interface-register-descriptions/Interrupt-Acknowledge-Register--GICC-IAR  
https://www.scs.stanford.edu/~zyedidia/docs/arm/armv8_baremetal.pdf  
https://stackoverflow.com/questions/66962053/why-kernels-arm64-vector-table-is-aligned-with-11  
https://github.com/raspberrypi/tools/blob/master/armstubs/armstub8.S  
https://github.com/rsta2/circle/blob/master/boot/config64.txt  
https://github.com/rsta2/circle/blob/master/boot/armstub/armstub8.S  
https://www.raspberrypi.org/app/uploads/2012/02/BCM2835-ARM-Peripherals.pdf  
https://github.com/Utsav-Agarwal/HobOS/blob/main/boot.S  
https://developer.arm.com/documentation/ddi0471/b/  
https://developer.arm.com/documentation/ihi0048/b/Programmers--Model/CPU-interface-register-descriptions/CPU-Interface-Control-Register--GICC-CTLR  
https://developer.arm.com/documentation/ihi0048/b/Programmers--Model/CPU-interface-register-descriptions/Interrupt-Priority-Mask-Register--GICC-PMR   
https://developer.arm.com/documentation/ihi0048/b/Programmers--Model/Distributor-register-descriptions/Distributor-Control-Register--GICD-CTLR  
https://developer.arm.com/documentation/ihi0048/b/Programmers--Model/Distributor-register-descriptions/Interrupt-Group-Registers--GICD-IGROUPRn  
https://developer.arm.com/documentation/ddi0601/2025-12/External-Registers/GICD-ITARGETSR-n---Interrupt-Processor-Targets-Registers  
[/writeup/raspberry5_baremetal_uart.md](/writeup/raspberry5_baremetal_uart.md)