Mastering Real-Time Operating Systems: From Bare-Metal to SafeRTOS

Real-time operating systems (RTOS) are the heart of modern embedded systems. Over the years, I've worked with various RTOS platforms—from bare-metal implementations to safety-certified systems. Here's what I've learned about choosing, implementing, and optimizing real-time systems.

The RTOS Landscape: Choosing the Right Foundation

When to Go Bare-Metal

Bare-metal development (no RTOS) makes sense when:

Ultra-low latency is critical (< 1 microsecond)
The application is simple with predictable execution
Memory footprint must be minimal
You need complete control over timing

Example: Safety Monitor Implementation

// Bare-metal safety monitor with precise timing
void safety_monitor_task(void) {
    static uint32_t last_check = 0;
    uint32_t current_time = get_microseconds();

    // Guaranteed 100μs execution every 1ms
    if (current_time - last_check >= 1000) {
        check_emergency_stop();      // 20μs
        validate_sensors();           // 30μs
        update_watchdog();            // 10μs
        check_communication();        // 25μs

        last_check = current_time;
    }
}

When You Need an RTOS

An RTOS becomes necessary when:

Multiple concurrent tasks with different priorities
Complex timing requirements
Need for inter-task communication
Resource sharing between tasks
Stack management and task isolation

FreeRTOS: The Workhorse

FreeRTOS has been my go-to RTOS for non-safety-critical applications. Its small footprint, widespread adoption, and MIT license make it ideal for many projects.

Task Management

// FreeRTOS task structure
void sensor_task(void *params) {
    const TickType_t freq = pdMS_TO_TICKS(10);  // 10ms period
    TickType_t last_wake = xTaskGetTickCount();

    for (;;) {
        // Read sensors
        sensor_data_t data = read_all_sensors();

        // Send to processing task via queue
        xQueueSend(sensor_queue, &data, portMAX_DELAY);

        // Precise periodic execution
        vTaskDelayUntil(&last_wake, freq);
    }
}

void processing_task(void *params) {
    sensor_data_t data;

    for (;;) {
        // Block until data available
        if (xQueueReceive(sensor_queue, &data, portMAX_DELAY)) {
            process_sensor_data(&data);

            // Signal completion
            xSemaphoreGive(processing_done_sem);
        }
    }
}

Priority Configuration

At TTControl, working with NXP iMX8QM's multiple Cortex-M4 cores, task priorities are critical:

// Priority hierarchy (higher number = higher priority)
#define PRIORITY_SAFETY_MONITOR    (configMAX_PRIORITIES - 1)  // 7
#define PRIORITY_COMMUNICATION     (configMAX_PRIORITIES - 2)  // 6
#define PRIORITY_SENSOR_PROCESSING (configMAX_PRIORITIES - 3)  // 5
#define PRIORITY_HMI_UPDATE        (configMAX_PRIORITIES - 4)  // 4
#define PRIORITY_LOGGING           (tskIDLE_PRIORITY + 1)      // 1

xTaskCreate(safety_monitor_task, "Safety",
            512, NULL, PRIORITY_SAFETY_MONITOR, NULL);

xTaskCreate(communication_task, "Comm",
            1024, NULL, PRIORITY_COMMUNICATION, NULL);

Resource Protection

// Mutex for shared resource access
static SemaphoreHandle_t spi_mutex = NULL;

void init_spi_protection(void) {
    spi_mutex = xSemaphoreCreateMutex();
    configASSERT(spi_mutex != NULL);
}

ErrorCode_t spi_transfer(uint8_t *tx, uint8_t *rx, size_t len) {
    ErrorCode_t result = ERROR_TIMEOUT;

    // Acquire mutex with timeout
    if (xSemaphoreTake(spi_mutex, pdMS_TO_TICKS(100)) == pdTRUE) {
        // Critical section - exclusive SPI access
        result = spi_transfer_blocking(tx, rx, len);

        xSemaphoreGive(spi_mutex);
    }

    return result;
}

SafeRTOS: Certified Safety

SafeRTOS is FreeRTOS's safety-certified sibling, qualified to:

IEC 61508 (SIL 3)
ISO 26262 (ASIL D)
EN 50128 (SIL 4)
DO-178C (Level A)

Key Differences from FreeRTOS

1. Deterministic Memory

// SafeRTOS requires static allocation
static StaticTask_t safety_task_tcb;
static StackType_t safety_task_stack[SAFETY_STACK_SIZE];

TaskHandle_t safety_handle = xTaskCreateStatic(
    safety_task_function,
    "Safety",
    SAFETY_STACK_SIZE,
    NULL,
    SAFETY_PRIORITY,
    safety_task_stack,
    &safety_task_tcb
);

// No dynamic allocation allowed!

2. Return Value Checking

// SafeRTOS - all return values must be checked
BaseType_t result = xQueueSend(queue, &data, timeout);
if (result != pdPASS) {
    log_error(ERROR_QUEUE_SEND_FAILED);
    trigger_safety_action();
    return ERROR_COMMUNICATION_FAILURE;
}

// FreeRTOS - return checking is optional (but recommended)
xQueueSend(queue, &data, timeout);  // Might fail silently

3. Restricted API Surface

SafeRTOS omits some FreeRTOS features:

Event groups (use binary semaphores instead)
Software timers (use tasks)
Co-routines (use tasks)
Stream buffers (use queues)

SafeRTOS Design Pattern

// Safety-critical task pattern
void safety_critical_task(void *params) {
    TickType_t last_wake = xTaskGetTickCount();
    const TickType_t period = pdMS_TO_TICKS(SAFETY_PERIOD_MS);

    for (;;) {
        BaseType_t status;

        // Read safety-critical input
        SafetyInput_t input;
        status = read_safe_input(&input);
        if (status != pdPASS) {
            handle_input_failure();
        }

        // Process with timeout guarantee
        SafetyOutput_t output;
        status = process_safety_logic(&input, &output);
        if (status != pdPASS) {
            enter_safe_state();
        }

        // Output with verification
        status = write_safe_output(&output);
        if (status != pdPASS) {
            trigger_emergency_stop();
        }

        // Pet the watchdog
        refresh_watchdog();

        // Precise timing
        vTaskDelayUntil(&last_wake, period);
    }
}

VxWorks & PikeOS: Aviation-Grade RTOS

At RT-RK, I developed drivers for VxWorks and PikeOS, targeting ARINC 653 partitioned systems for aviation.

ARINC 653 Partitioning

// PikeOS partition configuration
#define PARTITION_DRIVER    0
#define PARTITION_APP       1
#define PARTITION_MONITOR   2

// Strict temporal partitioning
// Each partition gets allocated time slots
// Memory is isolated - no shared access

// Inter-partition communication via ARINC 653 API
SAMPLING_PORT_ID_TYPE port_id;
MESSAGE_SIZE_TYPE msg_size = sizeof(HeartbeatMsg_t);
RETURN_CODE_TYPE ret;

CREATE_SAMPLING_PORT(
    "HEARTBEAT_OUT",      // Port name
    msg_size,             // Message size
    OUTPUT,               // Direction
    SAMPLING_PERIOD,      // Refresh period
    &port_id,            // Returned port ID
    &ret
);

if (ret == NO_ERROR) {
    HeartbeatMsg_t msg = { .counter = heartbeat_count };

    WRITE_SAMPLING_MESSAGE(
        port_id,
        (MESSAGE_ADDR_TYPE)&msg,
        msg_size,
        &ret
    );
}

VxWorks Real-Time Characteristics

VxWorks strengths:

Excellent interrupt latency (< 5μs on modern hardware)
Rich POSIX support
Mature networking stack
Extensive debugging tools

// VxWorks task creation
TASK_ID task_id = taskSpawn(
    "tSensor",                  // Name
    100,                        // Priority (0-255, lower is higher)
    VX_FP_TASK,                // Options (floating point)
    8192,                       // Stack size
    (FUNCPTR)sensor_task,      // Entry point
    0, 0, 0, 0, 0,            // Arguments
    0, 0, 0, 0, 0
);

if (task_id == TASK_ID_ERROR) {
    logMsg("Failed to create sensor task\n", 0,0,0,0,0,0);
}

Multi-Core RTOS Challenges

Working with NXP iMX8QM (multiple ARM Cortex-M4 cores) presents unique challenges:

Inter-Core Communication

// Shared memory region (uncached)
typedef struct {
    volatile uint32_t flag;
    volatile uint32_t sequence;
    uint8_t data[256];
} __attribute__((aligned(64))) SharedData_t;

static SharedData_t *shared_mem = (SharedData_t*)SHARED_MEM_BASE;

// Core 0: Producer
void core0_send_data(const uint8_t *data, size_t len) {
    // Wait for previous transfer
    while (shared_mem->flag != 0) {
        taskYIELD();
    }

    // Write data
    memcpy((void*)shared_mem->data, data, len);
    shared_mem->sequence++;

    // Data memory barrier
    __DMB();

    // Signal ready
    shared_mem->flag = 1;

    // Trigger Core 1 interrupt
    trigger_inter_core_interrupt(CORE_1);
}

// Core 1: Consumer
void core1_isr(void) {
    BaseType_t higher_priority_woken = pdFALSE;

    if (shared_mem->flag == 1) {
        // Read data
        uint8_t buffer[256];
        memcpy(buffer, (void*)shared_mem->data, sizeof(buffer));

        // Clear flag
        shared_mem->flag = 0;

        // Process in task context
        xQueueSendFromISR(process_queue, buffer, &higher_priority_woken);
    }

    portYIELD_FROM_ISR(higher_priority_woken);
}

Symmetric vs Asymmetric Multiprocessing

AMP (Asymmetric): Each core runs its own RTOS instance

Better isolation
Independent failure domains
Complex inter-core communication

SMP (Symmetric): Single RTOS manages all cores

Simpler programming model
Better load balancing
Shared failure domain

Current TTControl project uses AMP for safety isolation.

RTOS Debugging Techniques

Stack Overflow Detection

// FreeRTOS stack checking
#define configCHECK_FOR_STACK_OVERFLOW 2

void vApplicationStackOverflowHook(TaskHandle_t task, char *task_name) {
    // Task stack overflow detected!
    log_critical("Stack overflow in task: %s", task_name);

    // Capture task state
    TaskStatus_t task_status;
    vTaskGetInfo(task, &task_status, pdTRUE, eInvalid);

    log_critical("  Stack high water: %lu", task_status.usStackHighWaterMark);

    // Enter safe mode
    enter_safe_mode();
}

Timing Analysis

// Measure task execution time
void sensor_task(void *params) {
    for (;;) {
        uint32_t start = get_cycle_counter();

        // Task work
        process_sensors();

        uint32_t cycles = get_cycle_counter() - start;
        uint32_t us = cycles_to_microseconds(cycles);

        if (us > MAX_EXECUTION_TIME_US) {
            log_warning("Task exceeded timing: %lu us", us);
        }

        vTaskDelay(pdMS_TO_TICKS(10));
    }
}

Best Practices from the Field

1. Always Configure Stack Sizes Appropriately

// Calculate stack size based on:
// - Local variables
// - Function call depth
// - Interrupt nesting
// - Library usage

#define TASK_STACK_SIZE_SAFETY     (2048)  // Safety-critical
#define TASK_STACK_SIZE_COMM       (4096)  // Network stack
#define TASK_STACK_SIZE_UI         (8192)  // Qt/GUI

2. Use Queues for Data Transfer, Semaphores for Synchronization

// Good: Queue for data
xQueueSend(data_queue, &sensor_reading, timeout);

// Good: Semaphore for event signaling
xSemaphoreGive(data_ready_sem);

// Bad: Don't use queues as semaphores
xQueueSend(event_queue, &dummy_data, 0);

3. Minimize ISR Duration

void UART_IRQHandler(void) {
    BaseType_t higher_priority_woken = pdFALSE;

    if (UART_GetStatus() & UART_RX_READY) {
        uint8_t byte = UART_ReadByte();

        // Just queue the data, process in task
        xQueueSendFromISR(uart_rx_queue, &byte, &higher_priority_woken);
    }

    portYIELD_FROM_ISR(higher_priority_woken);
}

4. Plan for Failure

// Always have fallback behavior
BaseType_t result = xTaskCreate(/*...*/);
if (result != pdPASS) {
    // Attempt with reduced stack size
    result = xTaskCreate(/*..., smaller_stack*/);

    if (result != pdPASS) {
        // Fall back to bare-metal operation
        enable_bare_metal_mode();
    }
}

The Future: RTOS Evolution

Emerging trends in RTOS development:

Rust-based RTOS (memory safety without overhead)
Virtualization (running multiple RTOS instances)
AI/ML integration (predictive scheduling)
Formal verification (mathematical proof of correctness)

Conclusion

Mastering RTOS development requires understanding both theory and practice. From choosing bare-metal for ultra-low latency to leveraging SafeRTOS for safety certification, each approach has its place.

The key lessons:

Know your requirements: Timing, safety, complexity
Choose appropriate tools: Bare-metal, FreeRTOS, SafeRTOS, commercial RTOS
Design for predictability: Deterministic behavior trumps average performance
Test thoroughly: RTOS bugs are subtle and timing-dependent

Real-time systems are the foundation of embedded engineering. Get them right, and everything else falls into place.

This knowledge comes from years of hands-on RTOS development across multiple platforms and industries. Whether you're building your first RTOS application or architecting a safety-critical multi-core system, the principles remain the same: predictability, reliability, and rigorous engineering.