Rigel: Difference between revisions
(→repo: add archzfs) |
(change structure of regular.py -> cron tasks. also replace deprecated osm.py -> ramusage.py) |
||
| Line 1: | Line 1: | ||
Server that runs all applications and manages all data. It does not run the router. | Server that runs all applications and manages all data. It does not run the router. | ||
== | == cron jobs == | ||
Use the script <code>./runtask.sh {name}</code> to lookup said task in <code>tasks</code>. Any possible task has a name that determines the executable file, the logfile, and the lockfile. Each task can have a language of either <code>sh</code> or <code>py</code>, with py execute <code>python3 {name}.py</code> and with sh <code>./{name}.sh</code>. Then, each task has an environment to be run in: an lxc container or <code>rigel</code> direclty. In this way, <code>runtask.sh</code> provides a standard framework to manage logs and locking (each task never runs multiple copies of itself) and have all tasks described by only their name. | |||
To execute multiple tasks at one, just use <code>./runtask.sh {name1} {name2} {name3} [...]</code>. This is used for multiple different tasks that download something from the internet. Instead of scheduling them each offset by some guessed interval<syntaxhighlight lang="bash"> | |||
04 03 * * * /home/user/runtask.sh yt | |||
04 04 * * * /home/user/runtask.sh repos | |||
54 04 * * * /home/user/runtask.sh rsseth | |||
16 05 * * * /home/user/runtask.sh yttmp | |||
</syntaxhighlight>Just schedule the start of the chain:<syntaxhighlight lang="bash"> | |||
04 03 * * * /home/user/runtask.sh yt repos rsseth yttmp | |||
</syntaxhighlight> | |||
==== spball ==== | |||
Sponsorblock all: <code>rsync</code>s the 1GB-sized <code>sponsorTimes.csv</code> through <code>[[perun]]</code> lxc container. Then runs the python script <code>/home/user/addcsv.py</code> in [[lada]] to read the <code>.csv</code> file and load it into the [[Lada#RAMdb|ram-based database]]. There was a C program before to filter out some data columns, which reduced the actual data by about 5x, but It is discontinued because it liked to crash with a malformed csv input, which would happen because the <code>sponsorTimes.csv</code> is malformed. Instead, the whole file is read line-by-line in python and a few error checks are implemented, allowing it to fail somewhat gracefully. The python script then enforces the key-constraint as it is different from the one in the source csv, namely: key(line_uuid) for the csv becomes key(start,end,video_id) for the database. This means some lines will be thrown away, and some side-effects, like updating of already-existing lines being inconsistent/non-idempotent, and slight rounding errors due to the floating-point types of durations should be expected. Consult the script itself for details. | |||
==== | ==== bwusage ==== | ||
Quarter-hourly synchronization through <code>ssh</code> from [[nyx]]. Extracts data as json with:<syntaxhighlight lang="shell"> | |||
ssh nyx vnstat -i usb0 --json f | ssh nyx vnstat -i usb0 --json f | ||
</syntaxhighlight>The <code>.json</code> file is then | </syntaxhighlight>The <code>.json</code> file is then converted to multiple files to summarize and sum on time, also see [[Lada#Stats|prepending to make it Javascript]]. | ||
== ramusage.py == | |||
Python script that monitors RAM usagfe and attempts to restart the <code>renderd</code> daemon. Now that [[Osm#Tirex|Tirex]] is in use, it should not have a memory leak. | |||
Revision as of 08:59, 8 January 2023
Server that runs all applications and manages all data. It does not run the router.
cron jobs
Use the script ./runtask.sh {name} to lookup said task in tasks. Any possible task has a name that determines the executable file, the logfile, and the lockfile. Each task can have a language of either sh or py, with py execute python3 {name}.py and with sh ./{name}.sh. Then, each task has an environment to be run in: an lxc container or rigel direclty. In this way, runtask.sh provides a standard framework to manage logs and locking (each task never runs multiple copies of itself) and have all tasks described by only their name.
To execute multiple tasks at one, just use ./runtask.sh {name1} {name2} {name3} [...]. This is used for multiple different tasks that download something from the internet. Instead of scheduling them each offset by some guessed interval
04 03 * * * /home/user/runtask.sh yt
04 04 * * * /home/user/runtask.sh repos
54 04 * * * /home/user/runtask.sh rsseth
16 05 * * * /home/user/runtask.sh yttmpJust schedule the start of the chain:
04 03 * * * /home/user/runtask.sh yt repos rsseth yttmpspball
Sponsorblock all: rsyncs the 1GB-sized sponsorTimes.csv through perun lxc container. Then runs the python script /home/user/addcsv.py in lada to read the .csv file and load it into the ram-based database. There was a C program before to filter out some data columns, which reduced the actual data by about 5x, but It is discontinued because it liked to crash with a malformed csv input, which would happen because the sponsorTimes.csv is malformed. Instead, the whole file is read line-by-line in python and a few error checks are implemented, allowing it to fail somewhat gracefully. The python script then enforces the key-constraint as it is different from the one in the source csv, namely: key(line_uuid) for the csv becomes key(start,end,video_id) for the database. This means some lines will be thrown away, and some side-effects, like updating of already-existing lines being inconsistent/non-idempotent, and slight rounding errors due to the floating-point types of durations should be expected. Consult the script itself for details.
bwusage
Quarter-hourly synchronization through ssh from nyx. Extracts data as json with:
ssh nyx vnstat -i usb0 --json fThe .json file is then converted to multiple files to summarize and sum on time, also see prepending to make it Javascript.
ramusage.py
Python script that monitors RAM usagfe and attempts to restart the renderd daemon. Now that Tirex is in use, it should not have a memory leak.
