Andre Medeiros
73e7927d3f
update scheduler to split account work
2023-03-16 10:01:58 -04:00
Andre Medeiros
7582c90468
change notification interval to 10 seconds
2022-11-10 12:05:57 -05:00
Andre Medeiros
5eabaebf45
batch size wants to be smaller for less lockign time, not higher
2022-11-05 21:13:05 -04:00
Andre Medeiros
b18ae59207
increase scheduler batch size
2022-11-05 21:08:49 -04:00
Andre Medeiros
77a6f67c7b
small tweaks
2022-11-05 20:01:35 -04:00
Andre Medeiros
70f48c87da
make connections to the correct redis url envs
2022-11-05 16:28:27 -04:00
André Medeiros
9341e08123
feat(ops): add distributed tracing ( #109 )
...
* tracing
* instrument redis
* add instrumentation around queue acks
* adjust backoff schedules for reddit
* set proper http client on reddit
* populate account last message ID on upsert
* pass last message id on upsert
* schedule accounts in a predictable order
2022-11-05 15:59:33 -04:00
Andre Medeiros
6cf7c54c86
upgrade rmq to v5
2022-11-01 19:02:25 -04:00
Andre Medeiros
420499c9bb
update scheduler
2022-11-01 13:14:33 -04:00
Andre Medeiros
646dc0dd34
implement account soft delete
2022-11-01 12:45:11 -04:00
Andre Medeiros
a441d2723e
better use of context
2022-10-31 22:33:11 -04:00
Andre Medeiros
f01d305c57
explicitly pass context
2022-10-31 21:49:34 -04:00
Andre Medeiros
d9f7689d27
one context per goroutine
2022-10-31 21:48:47 -04:00
Andre Medeiros
eb1d904f86
change query
2022-10-31 21:32:43 -04:00
Andre Medeiros
d468a6e88b
change how we iterate over accounts
2022-10-31 21:30:12 -04:00
Andre Medeiros
683edbb091
dial down live activities
2022-10-31 20:58:02 -04:00
Andre Medeiros
df0860dd05
increase consumers and make choice more direct
2022-10-31 17:33:41 -04:00
Andre Medeiros
00bd99b60e
add logging to the scheduler
2022-10-31 15:49:27 -04:00
Andre Medeiros
c9877879da
same timezone everywhere
2022-10-27 15:47:26 -04:00
Andre Medeiros
2c31a0df84
increase cron concurrent jobs
2022-10-27 10:09:40 -04:00
Andre Medeiros
030b73e493
increase live activity batch size
2022-10-26 22:55:08 -04:00
Andre Medeiros
c08de574e9
fix context usage in scheduler
2022-10-26 20:48:22 -04:00
Andre Medeiros
13bc85dc8d
report on live activities
2022-10-26 20:36:03 -04:00
Andre Medeiros
696f932baa
Live activities
2022-10-19 09:37:41 -04:00
Andre Medeiros
13af0d1b4a
use the right id field
2022-07-31 15:23:29 -04:00
Andre Medeiros
bb2f709259
fix batches
2022-07-31 15:12:31 -04:00
Andre Medeiros
1617bf88e1
less mutations on the accounts table
2022-07-31 15:07:14 -04:00
Andre Medeiros
f4fea41435
remove some load
2022-07-13 18:43:27 -04:00
Andre Medeiros
ae764077b4
increase batch size on scheduler
2022-07-13 17:01:04 -04:00
Andre Medeiros
80bae7ff9f
get string slice straight from command result
2022-07-13 16:24:51 -04:00
Andre Medeiros
a00e4405c7
tweak redis conns
2022-07-13 15:02:38 -04:00
Andre Medeiros
a3e69954c8
up the batch sizes
2022-07-13 12:39:37 -04:00
Andre Medeiros
2fe34ecc99
run in singleton mode
2022-07-13 12:21:12 -04:00
Andre Medeiros
e80579ed35
tweak batch sizes
2022-07-13 12:19:11 -04:00
André Medeiros
e8a8e5a5b3
chore: use reddit ids when enqueueing accounts ( #87 )
2022-07-13 11:50:13 -04:00
Andre Medeiros
ce944f92f7
lets prune accounts again
2022-06-30 09:56:38 -04:00
Andre Medeiros
8ae426a4de
do not prune accounts
2022-06-25 14:45:05 -04:00
Andre Medeiros
221de1d2b6
back to no singleton mode, now that we're stable
2022-06-04 10:29:33 -04:00
Andre Medeiros
d243cebab6
better iteration
2022-06-04 10:24:22 -04:00
Andre Medeiros
084407a5f2
calculate batch size properly
2022-06-04 10:23:45 -04:00
Andre Medeiros
b23a158471
parallelize scheduler better
2022-06-04 10:21:29 -04:00
Andre Medeiros
6ece4c784a
no singleton
2022-05-27 16:12:50 -04:00
Andre Medeiros
0938400dd7
Revert "try enqueue accounts in a loop instead"
...
This reverts commit 9b5829f549
.
2022-05-27 15:02:57 -04:00
Andre Medeiros
01ad9f3d1e
Revert "another experiment for scheduling"
...
This reverts commit c8dd76e1ab
.
2022-05-27 15:02:47 -04:00
Andre Medeiros
c8dd76e1ab
another experiment for scheduling
2022-05-27 14:55:59 -04:00
Andre Medeiros
9b5829f549
try enqueue accounts in a loop instead
2022-05-27 14:47:59 -04:00
Andre Medeiros
fb3f2d1433
more scheduler tweaks
2022-05-27 14:40:40 -04:00
Andre Medeiros
356402d631
Revert "tweak scheduler"
...
This reverts commit 38fef3c852
.
2022-05-27 14:39:12 -04:00
Andre Medeiros
38fef3c852
tweak scheduler
2022-05-27 14:33:46 -04:00
Andre Medeiros
eb853710c4
Add profiling to scheduler
2022-05-27 13:27:28 -04:00