Rigel

Server that runs all applications and manages all data. It does not run the router.

cron jobs

Use the script ./runtask.sh {name} to lookup said task in tasks. Any possible task has a name that determines the executable file, the logfile, and the lockfile. Each task can have a language of either sh or py, with py execute python3 {name}.py and with sh ./{name}.sh. Then, each task has an environment to be run in: an lxc container or rigel direclty. In this way, runtask.sh provides a standard framework to manage logs and locking (each task never runs multiple copies of itself) and have all tasks described by only their name.

To execute multiple tasks at one, just use ./runtask.sh {name1} {name2} {name3} [...]. This is used for multiple different tasks that download something from the internet. Instead of scheduling them each offset by some guessed interval

04 03 * * * /home/user/runtask.sh yt
04 04 * * * /home/user/runtask.sh repos
54 04 * * * /home/user/runtask.sh rsseth
16 05 * * * /home/user/runtask.sh yttmp

Just schedule the start of the chain:

04 03 * * * /home/user/runtask.sh yt repos rsseth yttmp

See Perun for more concrete task descriptions.

spball

Sponsorblock all: rsyncs the 1GB-sized sponsorTimes.csv through perun lxc container. Then runs the python script /home/user/addcsv.py in lada to read the .csv file and load it into the ram-based database. There was a C program before to filter out some data columns, which reduced the actual data by about 5x, but It is discontinued because it liked to crash with a malformed csv input, which would happen because the sponsorTimes.csv is malformed. Instead, the whole file is read line-by-line in python and a few error checks are implemented, allowing it to fail somewhat gracefully. The python script then enforces the key-constraint as it is different from the one in the source csv, namely: key(line_uuid) for the csv becomes key(start,end,video_id) for the database. This means some lines will be thrown away, and some side-effects, like updating of already-existing lines being inconsistent/non-idempotent, and slight rounding errors due to the floating-point types of durations should be expected. Consult the script itself for details.

bwusage

Quarter-hourly synchronization through ssh from nyx. Extracts data as json with:

ssh nyx vnstat -i usb0 --json f

The .json file is then converted to multiple files to summarize and sum on time, also see prepending to make it Javascript.

ramusage.py

Python script that monitors RAM usagfe and attempts to restart the renderd daemon. Now that Tirex is in use, it should not have a memory leak.