celery

Just Queue It!

Queue Everything & Delight Everyone.

What You Will Learn?

Fundamentals of task queue.

What Is A Task Queue?

Taks Queue is a mechanism to distribute work across machines/threads.

How Task Queue Works?

A Simple Queue

A Simple Python Process

Token Based Queue System

A task queue

Why Task Queues?

Running Tasks In Background.

Complete A Task With Retries.

Multi Processing.

Scheduling Periodic Work.

Task Queue Implementation.

Celery

It’s a task queue with focus on real-time processing, while also supporting task scheduling.

Getting Started

Install Celery.

Pick a broker & install it.

Create tasks & processing them.

Supported Brokers

Which Broker?

Low volume tasks - Any broker.

High volume tasks - RabbitMQ or Redis.

RabbitMQ

(Designed for messaging)

Advanced Routing.

Reliable delivery through acknowledgement.

Can push > 100k messages per second.

Redis

(In memory data store)

Lightweight.

Blazing fast.

A simple function

def add(x, y):
    return x + y

result = add(3, 4)

Tasks

# task.py
from celery import Celery

app = Celery('tasks', backend='amqp',
             broker='amqp://guest@localhost//')


@app.task()
def add(x, y):
    return x + y

# add tasks into queue
async_result = add.delay(3, 4)

Workers

celery worker --help  # all options

celery worker -A task -l info

Results

# check result
async_result.successful()
async_result.state
async_result.result

Canvas Workflow

add.s(2, 3)  # subtask - signature object

add.si(2, 3)  # immutable signature object

result = chain(add.s(2, 2), add.s(4))()  # chain

result = group(add.s(i, i) for i in range(10))()  # group

result = chord((add.s(i, i) for i in xrange(10)), xsum.s())()  # chord

Routing

# run tasks in queues

add.apply_async([1, 2], queue='add_queue')

sub.apply_async([3, 4], queue='sub_queue')


# run multiple workers

celery worker -l info -A tasks -Q add_queue

celery worker -l info -A tasks -Q sub_queue

Monitoring.

celery inspect ping

celery inspect active

celery inspect active_queues

Flower

Real-time monitor & web admin for Celery.