How to fix Mono crashing on Odroid XU4
Recently I’ve been noticing my Sonarr and Radarr applications behaving erratic (sometimes not responsive, sometimes not performing tasks, not searching or adding content, but at other times behaving totally fine). A quick look at the logs told me the applications were crashing and being restarted by systemctl, after a few crashes they seemed to stabilize.
Today I had some time to dig deeper into the issue. I had already searched the Internet for general issues, but it didn’t seem to be a widespread problem. I assumed it might have to do with the ARM architecture, but Raspberry Pi users didn’t seem to be having these issues.
In the past I had difficulty reproducing the issue, but today I was in luck, every time I tried to kick off the “Process Monitored Downloads” task in Radarr, it would start working on the task and then crash and restart. The core issue turned out to be oddly specific to the Odroid XU4 hardware.
The XU4 has eight CPU cores, four A7 cores running at 1.4GHz, and four A15 cores running at 2GHz.
Whenever the mono process moved from an A7 to an A15 core (or vice versa) the process crashed.
Since both Sonarr and Radarr are Mono applications, they were both affected. Pinning the applications to either the A15 or the A7 CPUs resolved the problem.
taskset --cpu-list can be used to change the CPU affinity of a process.
First look up the systemctl service files for Radarr and Sonarr (e.g. via systemctl status radarr ). Edit the service files by prefixing the ExecStart command with /usr/bin/taskset -c 0-3 (for the A7 cores) or /usr/bin/taskset -c 4-7 (for the A15 cores).
Then reload the systemctl files ( systemctl daemon-reload ) and restart the service.
Example:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | # systemctl status sonarr ● sonarr.service - Sonarr Daemon Loaded: loaded (/lib/systemd/system/sonarr.service; enabled; vendor preset: enabled) Active: active (running) since Fri 2022-01-28 14:26:53 UTC; 38min ago Main PID: 20338 (mono) CGroup: /system.slice/sonarr.service └─20338 /usr/bin/mono --debug /usr/lib/sonarr/bin/Sonarr.exe -nobrowser -data=/var/lib/sonarr # vim /lib/systemd/system/sonarr.service [Unit] Description=Sonarr Daemon After=network.target [Service] User=media Group=media UMask=0002 Type=simple ExecStart=/usr/bin/taskset -c 0-3 /usr/bin/mono --debug /usr/lib/sonarr/bin/Sonarr.exe -nobrowser -data=/var/lib/sonarr TimeoutStopSec=20 KillMode=process Restart=on-failure [Install] WantedBy=multi-user.target # systemctl daemon-reload # systemctl restart sonarr |
If the service files are managed via the package manager, you may want to create an systemctl override instead of editing the service file, so that the package manager doesn’t overwrite your changes:
1 2 3 4 5 | # systemctl edit sonarr [Service] ExecStart= ExecStart=/usr/bin/taskset -c 0-3 /usr/bin/mono --debug /usr/lib/sonarr/bin/Sonarr.exe -nobrowser -data=/var/lib/sonarr |
Now you probably want to see if it worked and want to know how to check which core a process is running on? There a re a few options:
htop: launch htop . Press <F2> , go to Columns , and add PROCESSOR from Available Columns. The CPUs are numbered 1-8 in htop (as opposed to 0-7 by the system).
ps: the PSR column can display the core a process is on. The CPUs are numbered 0-7 in ps.
1 2 3 | # ps -o pid,psr,comm -p 21130 PID PSR COMMAND 21130 0 mono |
taskset: can display the current affinity of a process.
1 2 | # taskset -c -p 21130 pid 21130's current affinity list: 0-3 |