Commit graph

452 commits

Author SHA1 Message Date
André Medeiros
b5d52f55aa Merge 2dec2ff90e into 0accbde48b 2023-06-08 20:49:45 +02:00
Andre Medeiros
0accbde48b use a locking mechanism for the scheduler instead of the singleton model in the cron lib 2023-05-20 09:47:38 -04:00
Andre Medeiros
c216161836 remove events 2023-05-12 12:17:52 -04:00
Andre Medeiros
2dec2ff90e close pubsub 2023-04-01 12:19:56 -04:00
Andre Medeiros
9709c3a099 cache lock release function 2023-04-01 12:07:48 -04:00
Andre Medeiros
1dd0c3a47d first pass at distributed lock 2023-04-01 11:57:28 -04:00
Andre Medeiros
65fc7a5298 fix watchers 2023-03-31 11:57:09 -04:00
Andre Medeiros
782f77a7c4 close resultset earlier in scheduler 2023-03-24 17:05:00 -04:00
Andre Medeiros
4a7173ea9d fix delete too 2023-03-24 13:38:27 -04:00
Andre Medeiros
119ec8cc52 fix query for watchers 2023-03-24 13:37:35 -04:00
Andre Medeiros
b6cf6b6687 pgx/v4 to pgx/v5 2023-03-24 13:12:09 -04:00
Andre Medeiros
eb7ab043ed check err on subreddit watcher too 2023-03-24 12:48:47 -04:00
Andre Medeiros
00f17b7622 now that we cleared things up, let's show all errors 2023-03-16 13:18:29 -04:00
Andre Medeiros
199643449d make sure we don't run jobs that are too old 2023-03-16 13:12:31 -04:00
Andre Medeiros
fc0d290735 make the scheduler better by reducing redis calls 2023-03-16 16:43:35 +00:00
Andre Medeiros
0cadd624cc do not use deprecated setex 2023-03-16 12:10:04 -04:00
Andre Medeiros
6d0140151b make sure we get all of it 2023-03-16 12:01:04 -04:00
Andre Medeiros
f878a463b7 make logs less chatty 2023-03-16 10:44:09 -04:00
Andre Medeiros
74a87fce34 update dequeue metric 2023-03-16 10:34:15 -04:00
Andre Medeiros
8dd6242bff mark workgroups as done 2023-03-16 10:13:26 -04:00
Andre Medeiros
88223e6155 fix logging names 2023-03-16 10:07:36 -04:00
Andre Medeiros
f9fd53610f handle empty candidate set for scheduling 2023-03-16 10:06:51 -04:00
Andre Medeiros
73e7927d3f update scheduler to split account work 2023-03-16 10:01:58 -04:00
Andre Medeiros
b0f7f6fccb have errors for too many requests too 2023-03-16 09:33:51 -04:00
Andre Medeiros
4ec7338090 [render skip] log median score 2022-12-16 13:38:42 -05:00
Andre Medeiros
b78a2df159 [render skip] fix api live activities 2022-11-18 08:58:40 -05:00
Andre Medeiros
3b9da79e6e [render skip] add development in favour of sandbox 2022-11-13 09:08:55 -05:00
Andre Medeiros
fd648cad3a [render skip] notif tester 2022-11-11 12:03:39 -05:00
Andre Medeiros
7582c90468 change notification interval to 10 seconds 2022-11-10 12:05:57 -05:00
Andre Medeiros
d6b967188c things 2022-11-09 14:19:14 -05:00
Andre Medeiros
0ca21ffcb4 handle oauth revoked errors in watchers 2022-11-09 14:10:45 -05:00
Andre Medeiros
e23d47f815 switch to oauth in subredddit about 2022-11-09 13:59:04 -05:00
Andre Medeiros
038db61798 switch subreddit posts url 2022-11-09 13:54:03 -05:00
Andre Medeiros
fc449fd7cf [render skip] set last message id on api properly 2022-11-05 21:25:06 -04:00
Andre Medeiros
5eabaebf45 batch size wants to be smaller for less lockign time, not higher 2022-11-05 21:13:05 -04:00
Andre Medeiros
b18ae59207 increase scheduler batch size 2022-11-05 21:08:49 -04:00
Andre Medeiros
42aa9e594c bump locks again 2022-11-05 20:59:22 -04:00
Andre Medeiros
e241f67888 make pool sizes customized 2022-11-05 20:35:09 -04:00
Andre Medeiros
447a21ba60 increase prefetch 2022-11-05 20:31:55 -04:00
Andre Medeiros
194feb312c increase poolsize a bit 2022-11-05 20:25:41 -04:00
Andre Medeiros
6627fd4102 reduce pool size 2022-11-05 20:16:45 -04:00
Andre Medeiros
ded67f570d tweak pool sizes and consumer count 2022-11-05 20:10:05 -04:00
Andre Medeiros
77a6f67c7b small tweaks 2022-11-05 20:01:35 -04:00
Andre Medeiros
70f48c87da make connections to the correct redis url envs 2022-11-05 16:28:27 -04:00
Andre Medeiros
4f200fd26a spam prevention 2022-11-05 16:15:04 -04:00
Andre Medeiros
af538e6f42 update reddit http client 2022-11-05 16:08:00 -04:00
André Medeiros
9341e08123 feat(ops): add distributed tracing (#109)
* tracing

* instrument redis

* add instrumentation around queue acks

* adjust backoff schedules for reddit

* set proper http client on reddit

* populate account last message ID on upsert

* pass last message id on upsert

* schedule accounts in a predictable order
2022-11-05 15:59:33 -04:00
Andre Medeiros
ee1d1d32d4 try again with a queue specific redis connection 2022-11-03 10:30:50 -04:00
Andre Medeiros
164652e2e4 explicitly use http2 2022-11-02 19:05:23 -04:00
Andre Medeiros
c6903d2b5f Revert "each consumer gets their reddit client"
This reverts commit 5714fbaad7.
2022-11-02 19:02:25 -04:00