As a data engineer, testing with production loads is critical to performance checking, as well as finding edge cases where your assumptions about what can be expected in the data are curb stomped and send you back to the drawing board to cry and think about what you’ve done.
Yeah we finally set up a workflow where we get production data available in a staging environment. This has saved a lot of trouble via “well it worked on my local where there were 100 records, but prod has 1037492 and it does not”
As one of the devops/sysadmin types, if we give access to prod data to preprod systems, they are now in audit scope and you have to harden them or we lose our insurance and compliance certs.
Obviously the solution is to build some system where everything works out, but it’s not as easy as “just give root to devs”.
As a data engineer, testing with production loads is critical to performance checking, as well as finding edge cases where your assumptions about what can be expected in the data are curb stomped and send you back to the drawing board to cry and think about what you’ve done.
Yeah we finally set up a workflow where we get production data available in a staging environment. This has saved a lot of trouble via “well it worked on my local where there were 100 records, but prod has 1037492 and it does not”
deleted by creator
As one of the devops/sysadmin types, if we give access to prod data to preprod systems, they are now in audit scope and you have to harden them or we lose our insurance and compliance certs.
Obviously the solution is to build some system where everything works out, but it’s not as easy as “just give root to devs”.