I honestly don't know the exact time, I left it running for quite some time. I also didn't run it in one go. I started out with a too-small time window so I ended up running 3-4 batches in total to get all the data.
I made some updates to the script, see above.
The optimization trick from @crokkon made a huge difference. I didn't re-run the whole thing, but I compared the speed, running with max_batch_size set to 50 or as initially, without it. It's a speed boost somewhere from 10-20 times.
I honestly don't know the exact time, I left it running for quite some time. I also didn't run it in one go. I started out with a too-small time window so I ended up running 3-4 batches in total to get all the data.
I made some updates to the script, see above.
The optimization trick from @crokkon made a huge difference. I didn't re-run the whole thing, but I compared the speed, running with max_batch_size set to 50 or as initially, without it. It's a speed boost somewhere from 10-20 times.
Thank you I saw that improvement. But still it takes a long time hahaha I didn't know :)