Docker Queue

Laravel Docker Queue with Supervisor
Categories:
Tags:

We can easily to run the php artisan queue:work to run the queue job from the Laravel app by using supervisor to keep our queue:work processes running.

php artisan queue:work --queue=high,low

But we might want to run these jobs on the docker container instead of running the queue command directly.

File structure for supervisor on docker

We will use the docker compose to build the whole queue supervisor environment. The following file structure is the whole structure that we want to build for the entire environment.

- docker-compose.yaml
- resources
    - docker
        - dockerfile
            - supervisor.dockerfile
        - supervisor
            - conf.d
                - laravel-worker.conf

PHP 8 Supervidor Docker File

Because the queue job needs to run the entire Laravel App, so we need the entire same PHP docker environment that you run on the docker.

We need to access the redis queue database or the mysql, postgresql databases on the queue process. So the entire docker environment need to have ability to access all these database.

We will install the supervisor depending on the following requirement.

  • php
  • mysql or postgresql
  • redis
# resources/docker/dockerfile/supervisor.dockerfile
FROM php:8-fpm

RUN apt-get update && apt-get install -y libpq-dev libpng-dev libzip-dev zip unzip curl
RUN docker-php-ext-install pgsql pdo_pgsql gd zip\
  && docker-php-ext-enable opcache

# Install redis
RUN pecl install -o -f redis \
  &&  rm -rf /tmp/pear \
  &&  docker-php-ext-enable redis

# Install Supervisor
RUN apt-get install -y supervisor

RUN mkdir -p "/etc/supervisor/logs"

CMD ["/usr/bin/supervisord", "-n", "-c",  "/etc/supervisor/supervisord.conf"]

Laravel supervisor conf

We need to create the Laravel supervisor conf file to setting which command that we want to run.

php /var/www/html/laravel_app/artisan queue:work --queue=high,normal,low --sleep=30 --tries=3 --max-time=3600

We can decide how many process that we want to run on the same time.

numprocs=3

We can store the supervisor process log to our Laravel App folder

stdout_logfile=/var/www/html/laravel_app/storage/logs/worker.log

Here is the whole laravel supervisor conf that stores at the resources/docker/supervisor/conf.d/laravel-worker.conf

; resources/docker/supervisor/conf.d/laravel-worker.conf
[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /var/www/html/laravel_app/artisan queue:work --queue=high,normal,low --sleep=30 --tries=3 --max-time=3600
autostart=true
autorestart=true
stopasgroup=true
killasgroup=true
numprocs=3
redirect_stderr=true
stdout_logfile=/var/www/html/laravel_app/storage/logs/worker.log
stopwaitsecs=3600
stdout_logfile_maxbytes=5MB

Setting supervisor on the docker-compose.yaml

We can use the supervisor.dockerfile to run our environment. And volumes the entire Laravel App to the /var/www/html/laravel_app path.

And volumn the /resources/docker/supervisor/conf.d/ folder to the /etc/supervisor/conf.d/ path.

Set our entire supervisor docker container is depends on the postgresql and redis database.

# docker-compose.yaml
version: '3.9'

networks:
    local-proxy-network:
        driver: bridge

services:
    supervisor:
        container_name: local_supervisor
        build:
            context: resources/docker/dockerfile/
            dockerfile: supervisor.dockerfile
        image: local_supervisor:v1.0
        volumes:
            - '.:/var/www/html/laravel_app'
            - ./resources/docker/supervisor/conf.d/:/etc/supervisor/conf.d/
        working_dir: /var/www/html/laravel_app
        networks:
            - local-proxy-network
        restart: always
        depends_on:
            - postgresql
            - redis

Here is the entire docker-compose.yaml file

# docker-compose.yaml
version: '3.9'

networks:
    local-proxy-network:
        driver: bridge

services:
    postgresql:
        container_name: local_postgresql
        ports:
            - "5432:5432"
        image: "postgres:14.5"
        volumes:
            - ./storage/docker/postgresql/data:/var/lib/postgresql/data # persist data even if container shuts down
        environment:
            POSTGRES_USER: ${DB_USERNAME}
            POSTGRES_PASSWORD: ${DB_PASSWORD}
            POSTGRES_DB: ${DB_DATABASE}
        networks:
            - local-proxy-network
        restart: always
    redis:
        container_name: local_redis
        image: redis:6.2.7 # stable version
        ports:
            - "6379:6379"
        networks:
            - local-proxy-network
        restart: always
    supervisor:
        container_name: local_supervisor
        build:
            context: resources/docker/dockerfile/
            dockerfile: supervisor.dockerfile
        image: local_supervisor:v1.0
        volumes:
            - '.:/var/www/html/laravel_app'
            - ./resources/docker/supervisor/conf.d/:/etc/supervisor/conf.d/
        working_dir: /var/www/html/laravel_app
        networks:
            - local-proxy-network
        restart: always
        depends_on:
            - postgresql
            - redis

Run docker-compose

We can execute docker-compose up -d supervisor to running this supervisor container in background

# Run on the foreground
docker-compose up supervisor
# Run on the background
docker-compose up -d supervisor

Reference