Assuming the main modules are properly installed, we can proceed to the installation of workers. The worker can be installed on the same machine as the main modules, but for better measurement accuracy and scalability, it is recommended to install workers on separate (physical) machines.

Currently, the isolate sandbox still requires cgroups v1. Most distributions are forcing v2 in their kernels now, and it is imperative to make sure that your system is running with cgroups v1. You can check it by executing mount | grep cgroup and looking for cgroup entries (there should be multiple ones with different subsystems, e.g., cpu, memory, etc.). If you see only one entry with cgroup2 type, you are running with cgroups v2, and you need to switch to v1. The easiest way is to add SYSTEMD_CGROUP_ENABLE_LEGACY_FORCE=1 systemd.unified_cgroup_hierarchy=0 to kernel parameters in your bootloader (e.g., GRUB) and reboot. If that does not work, look for instructions for your distribution on how to switch to cgroups v1.

If you want to use disk quotas (recommended), you will need an FS that supports quotas. You will probably need to enable them in fstab, adding usrquota,grpquota to mount parameters (do not forget to remount/reboot). After that, you should install the quota tool and activate quotas (something like this):

# yum install quotatool quota
# mount -o remount /
# quotacheck -mavug
# quotaon --all

Similarly to the main modules, you can install the worker and cleaner from RPM packages (recommended) or from git repositories. The installation from RPM packages is straightforward:

$ dnf install recodex-worker recodex-cleaner

The worker is ready to be executed in multiple instances (i.e., one host may have multiple workers running). Each instance has a config file /etc/recodex/worker/config-%i.yml, where %i is the numeric ID of the worker (the first one has ID = 1). Make sure that you have a config file for each worker you want to start. However, it might be a good idea to configure one worker first, make sure it is running properly, and then use the first config as a template for others.

If you are managing many workers, a macro-preprocessing tool may be useful to manage their configurations. Each instance needs to be enabled and started (after the config file is ready) as (replace 1 with other IDs for other workers):

# systemctl enable recodex-worker@1
# systemctl start recodex-worker@1
# systemctl status recodex-worker@1

Before starting the worker service, edit the config file first. The worker-id (and optionally) worker-description distinguish individual workers in case you run multiple workers on the same machine. It is highly recommended that these config IDs match the IDs (numbers) of systemd services.

The worker needs the broker and special access to the core module (i.e., its worker-files interface) to operate. Update broker-uri so it matches your broker location and port designated to workers. The file-managers structure configures the core API module access (hostname has to be set to https URL pointing to the API, and HTTP auth credentials must match the credentials set in the workerFiles section of the core configuration).

Create worker(s) working directory (e.g., /var/recodex-worker-wd) and cache directory (e.g., /var/recodex-worker-cache), and set their paths to working-directory and file-cache > cache-dir properties respectively. Both directories must be owned (and writeable) by the user under which the worker runs (typically recodex). If you run multiple workers on one machine, these directories may be shared (recommended), and each worker creates its own subdirectories as needed. On the other hand, multiple workers should not share the log, so update logger > file so it holds a unique file for each worker.

The hwgroup holds an ID of a hardware group -- a group of workers with the same capabilities (i.e., running on the same hardware with the same system configuration). This is by default set to the only hwgroup present in init fixtures used to initialize the database for the first time. Optionally, you might want to restrict the runtime environments the worker supports (headers > env) based on which environments you plan to set up in the next step. The runtime IDs correspond to IDs in the database.

Based on supported runtime environments, it might be necessary to update the configuration of the sandbox (limits structure) -- namely, the pre-set environmental variables (environ-variable) and mapped directories (bound-directories). For instance, when using java runtime, a mapping for JDK directory needs to be added to bound-directories (src refers to directory on the real file systems, dst specifies, where the directory will be mounted in the sandbox), and JAVA_HOME environmental variable is needed with the same value as the destination (dst) of the mapped directory. We suggest proceeding with extra care when setting PATH and LD_LIBRARY_PATH environmental variables, as they may apply to multiple runtimes.

Once you complete your worker/environments debugging, it is recommended to set cleanup-submission to true, so that the worker removes old data from its working directory when each evaluation concludes (otherwise, all evaluation artifacts will linger there).

The worker uses sandbox isolate. The sandbox is installed automatically if you installed the worker as an RPM package (otherwise, you need to compile it manually). It has a configuration in /etc/isolate/default.cf. If you are running multiple workers (or other isolated services) on one machine, it might be a good idea to configure sandbox CPU affinity here, so that individual workers will not share CPU cores. For example:

box1.cpus = 0
box2.cpus = 1
box3.cpus = 2-7

Will configure box1 and box2 (which correspond to workers with IDs 1 and 2) as single-core boxes bound to the first and the second CPU core, and box3 will be a multi-core box occupying another six cores.

For greater precision, it is better not to utilize the entire CPU (all cores). Furthermore, we recommend turning off hyperthreading or multithreading features. The best setup is when one sandbox occupies one CPU socket alone, but that might be a waste if you are using CPU dies with many cores (consider that when planning the purchase of your hardware).

The cleaner module removes old records from the worker cache. It is a separate module since there may be multiple workers using the same cache.

After installing, edit the /etc/recodex/cleaner/config.yml. The most important parameter is cache-dir, which holds the path to the worker cache directory (i.e., /var/recodex-worker-cache if you followed the previous instructions). Additionally, you may set the expiration time and logging properties.

Although recodex-cleaner is also registered as a systemd service, you need to enable and start the recodex-cleaner.timer service, which runs the cleaner periodically (once a day). An alternative is to add a cleaner to cron, but the systemd timer service is the recommended way.

• Back to main Installation page
• Next: Setting up runtime environments