Commit graph

246 commits

Author SHA1 Message Date
Andre Medeiros
c4a0704883 enqueue directly from redis 2021-07-09 01:05:36 -04:00
Andre Medeiros
4a92b90df9 limit the amount of accounts we batch at any given time 2021-07-09 00:50:53 -04:00
Andre Medeiros
60e18c37f7 be more aggressive with scheduling 2021-07-09 00:27:50 -04:00
Andre Medeiros
961e41094d set the key only if it doesn't exist, otherwise it will never expire 2021-07-09 00:11:05 -04:00
Andre Medeiros
fb9741f7e0 make sure consumers always have plenty of work to do 2021-07-08 23:56:03 -04:00
Andre Medeiros
5cd3eeab09 fix delivery ack in jobs 2021-07-08 23:50:34 -04:00
Andre Medeiros
b9373832eb actually remember last message id so we don't duplicate notifications 2021-07-08 23:29:53 -04:00
Andre Medeiros
3ce858927b do batches in redis 2021-07-08 23:12:50 -04:00
Andre Medeiros
b7aea89cfc Measure schedule time 2021-07-08 22:15:28 -04:00
Andre Medeiros
d65bab7970 rejig sql 2021-07-08 22:09:14 -04:00
Andre Medeiros
9c70cee166 fix error handling 2021-07-08 21:50:45 -04:00
Andre Medeiros
3d8492e757 change locking around 2021-07-08 21:44:06 -04:00
Andre Medeiros
83fb4cdcc7 increase gomaxprocs 2021-07-08 21:28:43 -04:00
Andre Medeiros
26a034a68b fix logic again 2021-07-08 21:20:49 -04:00
Andre Medeiros
09c42c379c set things right 2021-07-08 21:07:01 -04:00
Andre Medeiros
0f70725be5 enqueue by batches 2021-07-08 21:01:45 -04:00
Andre Medeiros
45576b4d16 batch in second intervals 2021-07-08 20:53:09 -04:00
Andre Medeiros
4d2554fba6 bump up workers and poll duration 2021-07-08 20:44:12 -04:00
Andre Medeiros
abf8823cf1 use .Err to invoke the command 2021-07-08 20:37:39 -04:00
Andre Medeiros
f956033e2d update last checked at 2021-07-08 20:33:35 -04:00
Andre Medeiros
aad2434d83 fix skipped accounts and locks 2021-07-08 20:26:01 -04:00
Andre Medeiros
c1f1d2a309 log info 2021-07-08 20:17:20 -04:00
Andre Medeiros
4719ee091e clear lock after job succeeds too 2021-07-08 20:14:12 -04:00
Andre Medeiros
eddf7b53d1 fix notification rates 2021-07-08 20:11:46 -04:00
Andre Medeiros
2b9da40594 prioritize by last time accounts were checked 2021-07-08 20:06:22 -04:00
Andre Medeiros
66c4c28cb4 update account enqueueing logic 2021-07-08 20:03:56 -04:00
André Medeiros
a3d9c85c1d Merge pull request #1 from christianselig/feature/reworked-workers
Reworked workers
2021-07-08 19:56:53 -04:00
Andre Medeiros
6a5444bffe fix tags 2021-07-08 19:55:14 -04:00
Andre Medeiros
dea42a40f6 stats where they belong yay 2021-07-08 19:26:15 -04:00
Andre Medeiros
8eeda61c38 fix Procfile 2021-07-08 19:04:47 -04:00
Andre Medeiros
280c7eb593 added new things 2021-07-08 19:03:46 -04:00
Andre Medeiros
8455d8941c increase connection lifetime 2021-07-07 22:59:19 -04:00
Andre Medeiros
15c543bb62 tweak db connections 2021-07-07 22:58:21 -04:00
Andre Medeiros
0db6090f3e ease up on the stats rate 2021-07-07 22:54:26 -04:00
Andre Medeiros
bfa1342c62 fix reused measuring 2021-07-07 22:44:46 -04:00
Andre Medeiros
8fb6adfae2 maybe measure idle time better 2021-07-07 22:41:27 -04:00
Andre Medeiros
e56483dbb7 build up space for GC and main thread/statsd 2021-07-07 22:34:57 -04:00
Andre Medeiros
ff6bbf2a6f try using default client 2021-07-07 22:25:57 -04:00
Andre Medeiros
b55bb4839b measure reddit connections 2021-07-07 22:19:02 -04:00
Andre Medeiros
9156b7f07d measure dequeue time 2021-07-07 21:58:36 -04:00
Andre Medeiros
67ae62b765 add heroku metrics 2021-07-07 21:44:52 -04:00
Andre Medeiros
a1f0d73fc6 import fastparser, increase connectino count to 128 2021-07-07 21:01:54 -04:00
Andre Medeiros
827e61f3f9 change how we select accounts 2021-07-07 18:47:59 -04:00
Andre Medeiros
9b73c74486 fix expiry 2021-07-07 18:46:12 -04:00
Andre Medeiros
809b075804 fix float issue 2021-07-07 18:44:14 -04:00
Andre Medeiros
69e7cfea71 add some precision to last checked 2021-07-07 18:37:10 -04:00
Andre Medeiros
a929de67d4 postgres does not like being passed backoff 2021-07-07 18:12:45 -04:00
Andre Medeiros
ec7bb411fd measure if we are falling behind 2021-07-07 18:09:29 -04:00
Andre Medeiros
3b9a1ecd3d bump to 6 threads per cpu 2021-07-07 18:04:24 -04:00
Andre Medeiros
725ad8bf93 release transaction before sleeping in worker 2021-07-07 17:51:27 -04:00