Docker Queue
Laravel Docker Queue with Supervisor
Queue
Laravel Queue
Categories:
Running The Queue Worker
Note that once the queue:work command has started, it will continue to run until it is manually stopped or you close your terminal
php artisan queue:work
Queue workers are long-lived processes and store the booted application state in memory. During your deployment process, be sure to RESTART your queue workers.
php artisan queue:restart
Auto reload the latest code
When using the queue:listen command, you DON’T have to manually restart the worker when you want to reload your updated code or reset the application state; however, this command is significantly less efficient than the queue:work command:
php artisan queue:listen
Queue work options
Specifying The Connection & Queue
php artisan queue:work redis
# .env
QUEUE_CONNECTION=redis
Processing A Specified Number Of Jobs
The --max-jobs option may be used to instruct the worker to process the given number of jobs and then exit.
php artisan queue:work --max-jobs=1000
Processing Jobs For A Given Number Of Seconds
The --max-time option may be used to instruct the worker to process jobs for the given number of seconds and then exit.
php artisan queue:work --max-time=3600
Worker Sleep Duration
When jobs are available on the queue, the worker will keep processing jobs with no delay in between them. However, the sleep option determines how many seconds the worker will “sleep” if there are no new jobs available.
php artisan queue:work --sleep=3
Job Expirations & Timeouts
Job Expiration
if the value of retry_after is set to 90, the job will be released back onto the queue if it has been processing for 90 seconds without being released or deleted
// config/queue.php
'connections' => [
'redis' => [
'driver' => 'redis',
// Queue connection
'connection' => 'queue',
// Queue name
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
'after_commit' => false,
],
],
The queue:work Artisan command exposes a --timeout option. By default, the --timeout value is 60 seconds
If a job is processing for longer than the number of seconds specified by the timeout value, the worker processing the job will exit with an error.
php artisan queue:work --timeout=60
Dealing With Failed Jobs
After an asynchronous job has exceeded this number of attempts, it will be inserted into the failed_jobs database table.
php artisan queue:failed-table
php artisan migrate
Retry times
Specify the maximum number of times a job should be attempted using the --tries switch on the queue:work command
php artisan queue:work redis --tries=3
Wait for retry
Using the --backoff option, you may specify how many seconds Laravel should wait before retrying a job that has encountered an exception. By default, a job is immediately released back onto the queue so that it may be attempted again:
php artisan queue:work redis --tries=3 --backoff=3
/**
* The number of seconds to wait before retrying the job.
*
* @var int
*/
public $backoff = 3;
If you require more complex logic for determining the job’s backoff time, you may define a backoff method on your job class:
/**
* Calculate the number of seconds to wait before retrying the job.
*
* @return int
*/
public function backoff()
{
return 3;
}
You may easily configure “exponential” backoffs by returning an array of backoff values from the backoff method. In this example, the retry delay will be 1 second for the first retry, 5 seconds for the second retry, and 10 seconds for the third retry:
/**
* Calculate the number of seconds to wait before retrying the job.
*
* @return array
*/
public function backoff()
{
return [1, 5, 10];
}
Retrying Failed Jobs
View all of the failed jobs
php artisan queue:failed
Retrying Specific UUID Failed Jobs
php artisan queue:retry ce7bb17c-cdd8-41f0-a8ec-7b4fef4e5ece
Retry all of the failed jobs for a particular queue
php artisan queue:retry --queue=name
Retry all of your failed jobs
php artisan queue:retry all
Delete a specific failed job
php artisan queue:forget 91401d2c-0784-4f43-824c-34f94a33c24d
Delete all of your failed jobs
php artisan queue:flush
Pruning Failed Jobs
By default, all the failed job records that are more than 24 hours old will be pruned. If you provide the --hours option to the command, only the failed job records that were inserted within the last N number of hours will be retained
php artisan queue:prune-failed --hours=48
Disabling Failed Job Storage
# .env
QUEUE_FAILED_DRIVER=null
Clearing Jobs From Queues
php artisan queue:clear
php artisan queue:clear redis --queue=emails
Queue Priorities
dispatch((new Job)->onQueue('high'));
dispatch((new Job)->onQueue('medium'));
dispatch((new Job)->onQueue('low'));
To start a worker that verifies that all of the high queue jobs are processed before continuing to any jobs on the low queue, pass a comma-delimited list of queue names to the work command:
php artisan queue:work --queue=high,medium,low
Monitoring Your Queues
You should schedule the queue:monitor command to run every minute. The command accepts the names of the queues you wish to monitor as well as your desired job count threshold:
php artisan queue:monitor queue_connection:queue_name --max=100
php artisan queue:monitor redis:default,redis:deployments --max=100
Supervisor Configuration
In production, you need a way to keep your queue:work processes running. A queue:work process may stop running for a variety of reasons, such as an exceeded worker timeout or the execution of the queue:restart command.
For this reason, you need to configure a process monitor that can detect when your queue:work processes exit and automatically restart them
Install
sudo apt-get install supervisor
Configuring Supervisor
create a /etc/supervisor/conf.d/laravel-worker.conf file that starts and monitors queue:work processes:
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/laravel_app/artisan queue:work sqs --queue=high,medium,low --sleep=3 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
numprocs=3
redirect_stderr=true
stdout_logfile=/var/www/html/laravel_app/storage/logs/worker.log
stopwaitsecs=3600
Starting Supervisor
sudo supervisorctl reread
sudo supervisorctl update
sudo supervisorctl start laravel-worker:*