I once crashed a service because of my misunderstanding of one of the gotchas about how interfaces work in golang. Simplified version: https://go.dev/play/p/fbOcTx3B-a5 An unassigned instance of an interface is nil. However, once you assign an uninitialized implementation of the interface to an interface variable, the variable is no longer nil. In Golang(possibly in other languages as well), nil check for interface does not imply the underlying value(or implementation) of the interface is nil.

tldr; I split a long running job into smaller jobs but due to lack of proper prioritization of the jobs newly created numerous jobs starved the more important jobs. The Big Background Job We had a long running daily job. It would take 3-4 hours and we were fine with it since it was a background thing and we would run it at midnight. As the product grew and scaled we started to notice that it became longer and longer, 8-9 hours.