Commit graph

92 commits

Author SHA1 Message Date
Andre Medeiros
b992d23c03 use a locking mechanism for the scheduler instead of the singleton model in the cron lib 2023-06-08 17:33:22 -04:00
Andre Medeiros
782f77a7c4 close resultset earlier in scheduler 2023-03-24 17:05:00 -04:00
Andre Medeiros
b6cf6b6687 pgx/v4 to pgx/v5 2023-03-24 13:12:09 -04:00
Andre Medeiros
fc0d290735 make the scheduler better by reducing redis calls 2023-03-16 16:43:35 +00:00
Andre Medeiros
0cadd624cc do not use deprecated setex 2023-03-16 12:10:04 -04:00
Andre Medeiros
6d0140151b make sure we get all of it 2023-03-16 12:01:04 -04:00
Andre Medeiros
74a87fce34 update dequeue metric 2023-03-16 10:34:15 -04:00
Andre Medeiros
8dd6242bff mark workgroups as done 2023-03-16 10:13:26 -04:00
Andre Medeiros
88223e6155 fix logging names 2023-03-16 10:07:36 -04:00
Andre Medeiros
f9fd53610f handle empty candidate set for scheduling 2023-03-16 10:06:51 -04:00
Andre Medeiros
73e7927d3f update scheduler to split account work 2023-03-16 10:01:58 -04:00
Andre Medeiros
7582c90468 change notification interval to 10 seconds 2022-11-10 12:05:57 -05:00
Andre Medeiros
5eabaebf45 batch size wants to be smaller for less lockign time, not higher 2022-11-05 21:13:05 -04:00
Andre Medeiros
b18ae59207 increase scheduler batch size 2022-11-05 21:08:49 -04:00
Andre Medeiros
77a6f67c7b small tweaks 2022-11-05 20:01:35 -04:00
Andre Medeiros
70f48c87da make connections to the correct redis url envs 2022-11-05 16:28:27 -04:00
André Medeiros
9341e08123 feat(ops): add distributed tracing (#109)
* tracing

* instrument redis

* add instrumentation around queue acks

* adjust backoff schedules for reddit

* set proper http client on reddit

* populate account last message ID on upsert

* pass last message id on upsert

* schedule accounts in a predictable order
2022-11-05 15:59:33 -04:00
Andre Medeiros
6cf7c54c86 upgrade rmq to v5 2022-11-01 19:02:25 -04:00
Andre Medeiros
420499c9bb update scheduler 2022-11-01 13:14:33 -04:00
Andre Medeiros
646dc0dd34 implement account soft delete 2022-11-01 12:45:11 -04:00
Andre Medeiros
a441d2723e better use of context 2022-10-31 22:33:11 -04:00
Andre Medeiros
f01d305c57 explicitly pass context 2022-10-31 21:49:34 -04:00
Andre Medeiros
d9f7689d27 one context per goroutine 2022-10-31 21:48:47 -04:00
Andre Medeiros
eb1d904f86 change query 2022-10-31 21:32:43 -04:00
Andre Medeiros
d468a6e88b change how we iterate over accounts 2022-10-31 21:30:12 -04:00
Andre Medeiros
683edbb091 dial down live activities 2022-10-31 20:58:02 -04:00
Andre Medeiros
df0860dd05 increase consumers and make choice more direct 2022-10-31 17:33:41 -04:00
Andre Medeiros
00bd99b60e add logging to the scheduler 2022-10-31 15:49:27 -04:00
Andre Medeiros
c9877879da same timezone everywhere 2022-10-27 15:47:26 -04:00
Andre Medeiros
2c31a0df84 increase cron concurrent jobs 2022-10-27 10:09:40 -04:00
Andre Medeiros
030b73e493 increase live activity batch size 2022-10-26 22:55:08 -04:00
Andre Medeiros
c08de574e9 fix context usage in scheduler 2022-10-26 20:48:22 -04:00
Andre Medeiros
13bc85dc8d report on live activities 2022-10-26 20:36:03 -04:00
Andre Medeiros
696f932baa Live activities 2022-10-19 09:37:41 -04:00
Andre Medeiros
13af0d1b4a use the right id field 2022-07-31 15:23:29 -04:00
Andre Medeiros
bb2f709259 fix batches 2022-07-31 15:12:31 -04:00
Andre Medeiros
1617bf88e1 less mutations on the accounts table 2022-07-31 15:07:14 -04:00
Andre Medeiros
f4fea41435 remove some load 2022-07-13 18:43:27 -04:00
Andre Medeiros
ae764077b4 increase batch size on scheduler 2022-07-13 17:01:04 -04:00
Andre Medeiros
80bae7ff9f get string slice straight from command result 2022-07-13 16:24:51 -04:00
Andre Medeiros
a00e4405c7 tweak redis conns 2022-07-13 15:02:38 -04:00
Andre Medeiros
a3e69954c8 up the batch sizes 2022-07-13 12:39:37 -04:00
Andre Medeiros
2fe34ecc99 run in singleton mode 2022-07-13 12:21:12 -04:00
Andre Medeiros
e80579ed35 tweak batch sizes 2022-07-13 12:19:11 -04:00
André Medeiros
e8a8e5a5b3 chore: use reddit ids when enqueueing accounts (#87) 2022-07-13 11:50:13 -04:00
Andre Medeiros
ce944f92f7 lets prune accounts again 2022-06-30 09:56:38 -04:00
Andre Medeiros
8ae426a4de do not prune accounts 2022-06-25 14:45:05 -04:00
Andre Medeiros
221de1d2b6 back to no singleton mode, now that we're stable 2022-06-04 10:29:33 -04:00
Andre Medeiros
d243cebab6 better iteration 2022-06-04 10:24:22 -04:00
Andre Medeiros
084407a5f2 calculate batch size properly 2022-06-04 10:23:45 -04:00