Basic.publish: (406) PRECONDITION_FAILED - message size larger than configured max

[Muskanb]

I've encountered this error while passing arguments that are purely strings, and I'm doubtful that they can generate a size this large:
exception: Basic.publish: (406) PRECONDITION_FAILED - message size 179010654 is larger than configured max size 134217728

This is my defined task throwing this error :
@shared_task
def test(
arg1, arg2,arg3,
arg4=None, arg5='github', arg6="full", pullRequestId=None, baseBranchName='', checkRunId=None
):
These are either strings or integers

[Muskanb]

@auvipy It would be great if you could help :)

[hambro]

Had the same problem a year ago or so. The error only appeared if a pod had been running for a little bit of time. Tracked the bug down to the way Sentry patched into Celery, which appended a trace ID to the headers of the traces. At some point, we were sending tiny messages with 15Mb inane headers.
If you can reproduce it, see if you can get a hold of the actual payload to rabbitmq :)

[dexterionut]

Hey there! Any news on this? We are experiencing the same thing with the following version sentry-sdk=2.12.0 and celery=5.3.1

134217728 is 128MB, which IMO is already a very high size for a queue message for systems only sending integers/strings as parameters with the tasks.

@Muskanb Did you manage to find a solution?
@hambro how did you fix the problem on your side?

[Muskanb]

Tried looking into the message size but that was not causing the issue. For now we just added a retry mech:

def retrySafe(retries=3):
    def decorator(func):
        def wrapper(*args, **kwargs):
            for _ in range(retries):
                try:
                    func(*args, **kwargs)
                    return # If function execution is successful, return
                except Exception as e:
                    exceptionTrace = traceback.format_exc()
                    log.error(f"An error occurred: {exceptionTrace}. Retrying...")
            raise Exception(f"Function {func.__name__} failed after {retries} retries.")
        return wrapper
    return decorator

this decorator on wrapper function adds the task back to queue if it fails.

@retrySafe(retries=3)
def queueFunctionWithRetry():

[glumia]

Hey there, this seems to be a consequence of publisher confirms not being enabled by default with Celery.

What happens is that a previous .delay/.apply_async call that should have failed returns successfully, and then one gets the error in the following .delay/.apply_async that reuses the same rabbitmq connection.

You can find a reproducer for this at https://github.com/glumia/celery-rmq-precondition-failed.