This article is part of the CakeDC Advent Calendar 2025 (December 9th 2025)

Introduction: need of Concurrency

While offloading long-running tasks to an asynchronous queue solves the initial web request bottleneck, relying on a single queue worker introduces a new, serious point of failure and bottleneck. This single-threaded approach transfers the issue from the web server to the queue system itself.

Bottlenecks of Single-Worker Queue Processing

The fundamental limitation in the standard web request lifecycle is its synchronous, single-threaded architecture. This design mandates that a user's request must wait for all associated processing to fully complete before a response can be returned.

The Problem: Single-Lane Processing Imagine a queue worker as a single cashier at a very busy bank . Each item in the queue (the "job") represents a customer.

1. Job Blocking (The Long Transaction): If the single cashier encounters a customer with an extremely long or slow transaction (e.g., generating a massive report, bulk sending 100,000 emails, or waiting for a slow API), every other customer must wait for that transaction to complete.
2. Queue Backlog Accumulation: New incoming jobs (customers) pile up rapidly in the queue. This is known as a queue backlog. The time between a job being put on the queue and it starting to execute (Job Latency) skyrockets.
3. Real-Time Failure: If a job requires an action to happen now (like sending a password reset email), the backlog means that action is critically delayed, potentially breaking the user experience or application logic.
4. Worker Vulnerability and Downtime: If this single worker crashes (due to a memory limit or unhandled error) or is temporarily taken offline for maintenance, queue processing stops entirely. The application suddenly loses its entire asynchronous capability until the worker is manually restarted, resulting in a complete system freeze of all background operations.

To eliminate this bottleneck, queue consumption must be handled by multiple concurrent workers, allowing the system to process many jobs simultaneously and ensuring no single slow job can paralyze the entire queue.

Improved System Throughput and Reliability with Multiple Workers

While introducing a queue solves the initial issue of synchronous blocking, scaling the queue consumption with multiple concurrent workers is what unlocks significant performance gains and reliability for the application's background processes.

Key Benefits of Multi-Worker Queue Consumption

• Consistent, Low Latency: Multiple workers process jobs in parallel, preventing any single slow or heavy job (e.g., report generation) from causing a queue backlog. This ensures time-sensitive tasks, like password resets, are processed quickly, maintaining instant user feedback.
• Enhanced Reliability and Resilience: If one worker crashes, the other workers instantly take over** the remaining jobs. This prevents a complete system freeze and ensures queue processing remains continuous.
• Decoupling and Effortless Scaling: The queue facilitates decoupling. When background load increases, you simply deploy more CakePHP queue workers. This horizontal scaling is simple, cost-effective, and far more efficient than scaling the entire web server layer.

Workflows that Benefit from Multi-Worker Concurrency

These examples show why using multiple concurrent workers with the CakePHP Queue plugin (https://github.com/cakephp/queue) is essential for performance and reliability:

• Mass Email Campaigns (Throughput): Workers process thousands of emails simultaneously, drastically cutting the time for large campaigns and ensuring the entire list is delivered fast.
• Large Media Processing (Parallelism): Multiple workers handle concurrent user uploads or divide up thumbnail generation tasks. This speeds up content delivery by preventing one heavy image from blocking all others.
• High-Volume API Synchronization (Consistency): Workers ensure that unpredictable external API latency from one service doesn't paralyze updates to another. This maintains a consistent, uninterrupted flow of data across all integrations.

The Job

Lets say that you have the queue job like this:

<?php  
declare(strict_types=1);  

namespace App\Job;  

use Cake\Mailer\Mailer;  
use Cake\ORM\TableRegistry;  
use Cake\Queue\Job\JobInterface;  
use Cake\Queue\Job\Message;  
use Interop\Queue\Processor;  

/**  
 * SendBatchNotification job 
 */
class SendBatchNotificationJob implements JobInterface  
{  
    /**  
     * The maximum number of times the job may be attempted.     
     *     
     * @var int|null  
     */  
    public static $maxAttempts = 10;  

    /**
     * We need to set the shouldBeUnique to true to avoid race condition with multiple queue workers
     *
     * @var bool
     */
    public static $shouldBeUnique = true;

    /**  
     * Executes logic for SendBatchNotificationJob     
     *     
     * @param \Cake\Queue\Job\Message $message job message  
     * @return string|null  
     */  
    public function execute(Message $message): ?string  
    {  
        // 1. Retrieve job data from the message object  
        $data = $message->getArgument('data');  
        $userId = $data['user_id'] ?? null;  

        if (!$userId) {  
            // Log error or skip, but return ACK to remove from queue  
            return Processor::ACK;  
        }  

        try {  
            // 2. Load user and prepare email  
            $usersTable = TableRegistry::getTableLocator()->get('Users');  
            $user = $usersTable->get($userId);  

            $mailer = new Mailer('default');  
            $mailer  
                ->setTo($user->email)  
                ->setSubject('Your batch update is complete!')  
                ->setBodyString("Hello {$user->username}, \n\nThe recent batch process for your account has finished.");  

            // 3. Send the email (I/O operation that can benefit from concurrency)  
            $mailer->send();  

        } catch (\Exception $e) {  
            // If the email server fails, we can tell the worker to try again later  
            // The queue system will handle the delay and retry count.            
            return Processor::REQUEUE;  
        }  

        // Success: Acknowledge the job to remove it from the queue  
        return Processor::ACK;  
    }  
}

Setting $shouldBeUnique = true; in a CakePHP Queue Job class is crucial for preventing a race condition when multiple queue workers consume the same queue, as it ensures only one instance of the job is processed at any given time, thus avoiding duplicate execution or conflicting updates.

In another part of the application you have code that enqueues the job like this:

// In a Controller, Command, or Service Layer:
use Cake\ORM\TableRegistry;
use Cake\Queue\QueueManager;
use App\Job\SendBatchNotificationJob; // Our new Job class

// Find all users who need notification (e.g., 500 users)
$usersToNotify = TableRegistry::getTableLocator()->get('Users')->find()->where(['is_notified' => false]);

foreach ($usersToNotify as $user) {
    // Each loop iteration dispatches a distinct, lightweight job
    $data = [
        'user_id' => $user->id,
    ];

    // Dispatch the job using the JobInterface class name
    QueueManager::push(SendBatchNotificationJob::class, $data);
}

// Result: 500 jobs are ready in the queue.

By pushing 500 separate jobs, you allow 10, 20, or even 50 concurrent workers to pick up these small jobs and run the email sending logic in parallel, drastically reducing the total time it takes for all 500 users to receive their notification.

Implementing Concurrency with multiple queue workers

In modern Linux distributions, systemd is the preferred init and service manager. By leveraging User Sessions and the Lingering feature, we can run the CakePHP worker as a dedicated, managed service without needing root privileges for the process itself, offering excellent stability and integration.

SystemD User Sessions

Prerequisite: The Lingering User Session

For a service to run continuously in the background, even after the user logs out, we must enable the lingering feature for the user account that will run the workers (e.g., a service user named appuser).

Enabling Lingering:

Bash

sudo loginctl enable-linger appuser

This ensures the appuser's systemd user session remains active indefinitely, allowing the worker processes to survive server reboots and user logouts.

Creating the Systemd User Unit File

We define the worker service using a unit file, placed in the user's systemd configuration directory (~/.config/systemd/user/).

• File Location: ~appuser/.config/systemd/user/[email protected]
• Purpose of @: The @ symbol makes this a template unit. This allows us to use a single file to create multiple, distinct worker processes, which is key to achieving concurrency.

[email protected] Content:

Ini, TOML

[Unit]
Description=CakePHP Queue Worker #%i
After=network.target

[Service]
# We use the full path to the PHP executable
ExecStart=/usr/bin/php /path/to/your/app/bin/cake queue worker
# Set the current working directory to the application root
WorkingDirectory=/path/to/your/app
# Restart the worker if it fails (crashes, memory limit exceeded, etc.)
Restart=always
# Wait a few seconds before attempting a restart
RestartSec=5
# Output logs to the systemd journal
StandardOutput=journal
StandardError=journal
# Ensure permissions are correct and process runs as the user
User=appuser

[Install]
WantedBy=default.target

Achieving Concurrency (Scaling the Workers)

Concurrency is achieved by enabling multiple instances of this service template, distinguished by the suffix provided in the instance name (e.g., -1, -2, -3).

Reload and Start Instances: After creating the file, the user session must be reloaded, and the worker instances must be started and enabled:

Reload Daemon (as appuser):

Bash

systemctl --user daemon-reload

Start and Enable Concurrent Workers (as appuser): To run three workers concurrently:

Bash

# Start Worker Instance 1
systemctl --user enable --now [email protected]

# Start Worker Instance 2
systemctl --user enable --now [email protected]

# Start Worker Instance 3
systemctl --user enable --now [email protected]

Result: The system now has three independent and managed processes running the bin/cake queue worker command, achieving a concurrent processing pool of three jobs.

Monitoring and Management

systemd provides powerful tools for managing and debugging the worker pool:

Check Concurrency Status:

Bash

systemctl --user status 'cakephp-worker@*'

This command displays the status of all concurrent worker instances, showing which are running or if any have failed and been automatically restarted.

Viewing Worker Logs: All output is directed to the systemd journal:

Bash

journalctl --user -u 'cakephp-worker@*' -f

This allows developers to inspect errors and task completion messages across all concurrent workers from a single, centralized log.

Using systemd and lingering is highly advantageous as it eliminates the need for a third-party tool, integrates naturally with system logging, and provides reliable process management for a robust, concurrent task environment.

Summary

Shifting from a single worker to multiple concurrent workers is essential to prevent bottlenecks and system freezes caused by slow jobs, ensuring high reliability and low latency for asynchronous tasks.

One robust way to achieve this concurrency in CakePHP applications is by using Systemd User Sessions and template unit files (e.g., [email protected]) to easily manage and horizontally scale the worker processes.

This article is part of the CakeDC Advent Calendar 2025 (December 9th 2025)