YES! We found the problem! 2 weeks of troubleshooting... and the thing that everyone less expects to enable pin-pointing of the problem, is the recipe for success! Observability!
Decided to come share this here because I feel its one of the most valuable things for troubleshooting.