As we go on in our pursuit to use Joomla in never-seen-before use cases a few interesting problems came forth.
- A web-hooks implementation: Similar to how plugins can declare triggers they want to listen to, webhooks allow external applications to register URLs that can receive a JSON payload when an action happens in an application. Since there could be multiple webhooks, each webhook call is queued as a message and happens asynchronously.
- In Shika: Recalculating user progress in the background. When a new lesson is added to a course, Shika recalculates each enrolled user’s progress to take into account the newly added lesson. When there are a lot of lessons and a large number of users enrolled in the course, this recalculation cannot happen in real time. In this case, a recalculation job does this in the background once the lesson is saved.
The first thought, of course, was to create purpose specific tables and to manage using cron jobs. But being the engineers that we are, we felt the need to bring some of the best practices into Joomla. This is how the Joomla message queue was born!
We are happy to introduce our new infrastructure extension - TJ Queue, a generic job/message queue for Joomla. Messages queues are a way to delegate some processes to the background instead of performing them in a user blocking operation. It is also useful to move heavy processing to the background.
Any extension that wishes to use the background job queue, creates a consumer that performs the 'job'. A job could be as simple as sending an email or a notification, or some heavy-duty data crunching. The application can then 'produce' a message with the payload needed by the job. The TJ Queue cron job will fetch the messages from the queue and pass them on to the correct consumer to execute the job.
The extension uses php-enqueue as the underlying library. Currently the Amazon SQS and MySQL adapters are bundled with the extension, we will add support for more in the future.