Behind the Bug: Magento Errcode 28

When a Magento deployment fails during setup:upgrade, it's natural to assume the problem lies within Magento itself. That was certainly our initial thought when a recent staging deployment failed while updating modules and creating database triggers for inventory indexing.

The error appeared while Magento was processing a trigger referencing an Amasty Visual Merchandising table, which made it appear to be a module-related issue. However, after further investigation, it became clear that neither Magento nor Amasty was at fault. The real cause was a storage issue within MariaDB that prevented the database from completing the schema-related operations required during deployment.

This article walks through the symptoms, the investigation process, the root cause, and the steps taken to resolve the issue.

The error at hand

During the deployment, Magento failed while running setup:upgrade and creating MView subscriptions and indexer triggers. The error returned was:

SQLSTATE[HY000]: General error: 3 Error writing file './ddl_recovery.log'

Errcode: 28 "No space left on device"

The failing SQL operation was a CREATE TRIGGER statement against the inventory_source_item table. Part of the trigger logic referenced the following table:

INSERT IGNORE INTO amasty_merch_product_dynamic_category_msi_cl

Given that the error surfaced during the creation of an Amasty-related trigger, it was reasonable to suspect a problem with the extension or the generated SQL. However, focusing on the trigger itself quickly proved to be a distraction from the underlying issue.

Understanding the real error

The most important clue in the error message was not the trigger, but the reference to:

ddl_recovery.log

This file is used internally by MariaDB during DDL (Data Definition Language) operations such as CREATE TRIGGER, CREATE TABLE, ALTER TABLE, TRUNCATE TABLE, and OPTIMIZE TABLE. When MariaDB performs these operations, it writes recovery information that allows changes to be rolled back safely if the process is interrupted.

Once we focused on that detail, the error became much clearer. MariaDB was not reporting a problem with the trigger syntax. Instead, it was reporting that it could not write the recovery information required to operate safely. That shifted the investigation away from Magento and towards the database layer.

The first check was misleading

The first troubleshooting step was to verify available disk space using:

df -h

The results showed plenty of available storage, which initially seemed to contradict the error message. However, the deployment was running in a containerised environment, and the filesystem being inspected belonged to the application container rather than the MariaDB container.

To confirm where the database files were actually stored, we checked the MariaDB data directory:

SHOW VARIABLES LIKE 'datadir';

The result was:

/data/

At that point, an important detail emerged. The shell session we were using did not even contain a /data directory. This confirmed that we were inspecting the wrong environment and that the storage issue existed on the MariaDB container or its associated persistent volume rather than within the Magento application container.

Identifying the storage problem

With the focus shifted to the database volume, the next step was reviewing table sizes to identify any unusually large datasets.

One table immediately stood out:

webapi_logs

The webapi_logs table contained well over a million rows and had grown significantly larger than any other table in the database. The table originated from Web API logging and had been allowed to grow without any retention or cleanup process in place.

At this point, the deployment failure made sense. MariaDB had exhausted the available storage on its volume and was unable to write the recovery information required to create the deployment-related triggers.

Why TRUNCATE failed

The obvious solution appeared to be clearing the table using:TRUNCATE TABLE

TRUNCATE TABLE webapi_logs;

However, this immediately returned:

The table 'webapi_logs' is full

While this may seem counterintuitive, the explanation lies in how TRUNCATE works. Although the command removes all rows from a table, it's classified as a DDL operation rather than a DML operation. Before MariaDB can truncate the table, it still needs to write information to ddl_recovery.log.

Because the database volume had no remaining writable space, MariaDB was unable to perform even the operation intended to free up storage.

The temporary workaround

To create enough free space to recover the environment, we switched to deleting rows in batches:

DELETE FROM webapi_logs LIMIT 100000;

This was followed by larger batches as space became available:

DELETE FROM webapi_logs LIMIT 1000000;

Unlike TRUNCATE, DELETE is a DML operation and does not require the same DDL recovery process. As a result, MariaDB was able to execute these statements despite operating under severe storage pressure.

This approach reduced the size of the dataset and provided enough breathing room to continue working with the database.

Why deleting rows doesn’t reclaim disk space

It's important to understand that deleting rows and reclaiming storage are not necessarily the same thing.

The affected table was using the InnoDB storage engine. With InnoDB, deleting rows frees space within the table itself but does not automatically reduce the size of the underlying table file on disk. As a result, the operating system may show little or no reduction in actual disk usage even after a significant number of rows have been removed.

To return unused space to the filesystem, the table generally needs to be rebuilt using either:

TRUNCATE TABLE webapi_logs;

or:

OPTIMIZE TABLE webapi_logs;

Both operations recreate the table and reclaim unused disk space. However, they also require enough free storage to complete successfully, which is why creating breathing room through incremental deletes was necessary first.

The root cause

The root cause of the deployment failure was database storage exhaustion.

Magento happened to be performing schema-related work at the time, which is why the error surfaced during setup:upgrade. However, neither Magento nor the Amasty extension were responsible for the failure. MariaDB simply did not have sufficient free space available to write the files required for DDL recovery operations.

Viewed in that context, the error message was entirely accurate. The database had genuinely run out of available storage.

How to fix this error (step by step)

If you encounter the following error during a Magento deployment:

Error writing file './ddl_recovery.log'

Errcode: 28 "No space left on device"

The following troubleshooting process is worth following:

1. Verify where MariaDB stores its data using SHOW VARIABLES LIKE 'datadir';

2. Check storage usage on the database host, container, or persistent volume.

3. Review table sizes and identify unusually large tables.

4. Use batch DELETE statements to reduce data volume if DDL operations are already failing.

5. Once sufficient space is available, rebuild affected tables using TRUNCATE or OPTIMIZE to reclaim storage fully.

Preventing similar issues

While deleting rows resolved the immediate problem, the long-term solution is ensuring the database has sufficient capacity for ongoing maintenance and schema operations.

If Web API logging is enabled, it's worth reviewing whether the data being collected is genuinely required, particularly in staging environments. Implementing retention policies, scheduling automated cleanup jobs, and monitoring table growth can help prevent logging tables from consuming excessive storage over time.

Regular database housekeeping is often overlooked because large log tables rarely affect storefront functionality directly. However, as this incident demonstrated, they can eventually consume enough space to block deployments, upgrades, indexing operations, and schema changes.

Reflecting on the issue

What made this issue particularly misleading was that the failure appeared during a Magento deployment and involved a trigger that referenced a third-party extension. Everything initially pointed towards an application-level problem.

In reality, the issue was much lower down the stack. MariaDB had run out of space and could no longer perform the recovery operations required for DDL changes.

If you encounter Errcode: 28 alongside references to ddl_recovery.log, it's worth investigating database storage before spending too much time debugging Magento code or third-party modules. In our case, a single oversized logging table was enough to prevent deployments from completing successfully, and identifying that fact ultimately led to a straightforward resolution.