I once crashed a service because of my misunderstanding of one of the gotchas about how interfaces work in golang. Simplified version: https://go.dev/play/p/fbOcTx3B-a5 An unassigned instance of an interface is nil. However, once you assign an uninitialized implementation of the interface to an interface variable, the variable is no longer nil. In Golang(possibly in other languages as well), nil check for interface does not imply the underlying value(or implementation) of the interface is nil.

tldr: I sent a different ticket to a different user due to race condition Problem We had a consumer which would listen to transaction events and based on that would issue tickets to the corresponding user. Since that was something that could be done concurrently I designed a flow to set up a threadpool and assign the tasks. Problem was that ‘data’ was part of the object’s state and not a parameter so multiple task could see and modify the ‘data’ resulting in inconsistent behavior.

tldr; I split a long running job into smaller jobs but due to lack of proper prioritization of the jobs newly created numerous jobs starved the more important jobs. The Big Background Job We had a long running daily job. It would take 3-4 hours and we were fine with it since it was a background thing and we would run it at midnight. As the product grew and scaled we started to notice that it became longer and longer, 8-9 hours.