Fix the root bottleneck
Fix what makes something slow instead of hiding it with longer timeouts, retries or a degraded mode.
Fix the cause of slowness instead of masking it with a longer timeout, a retry or a degraded mode. Extra waiting can make the symptom go away while the cause stays and gets worse under load.
A workaround is acceptable only as a temporary step with a follow-up task to fix the cause.
Example
Package sql() writes sometimes hung. A longer timeout would have hidden it. The cause was connection starvation: every database connection a request had was held by a transaction waiting on the same lock, so the lock holder had none left to work with. The fix queues same-org calls before they take a connection:
/**
* Calls on the same database client also queue here first, so at most one of them waits on the
* advisory lock at a time. The lock holder reads and writes on the client's other connections
* (5 per request on Workers); with every connection held by a transaction waiting for the lock,
* it would get none and the run would hang.
*/Cold starts were handled the same way. Rather than keep workers warm, Minion trimmed what loads at startup. Embedding libraries now load on first use, which cut the worker bundle from 6.42 to 4.97 MiB and startup CPU from 73 to 63 ms. Separate work cut the database round trips for one execute call from 14 to 9.
In your packages
If a job is close to its time limit, find out what it waits on before asking for more time. Batch the API calls, page through results, or split the work across runs. See Runtime and limits and Heavy work .