db: fix dead lock in zmDbQueue::stop()

Notifying `mCondition` without taking the lock causes a race condition
in ::process() between checking `mTerminate` and waiting for the
`mCondition`, which causes a dead lock. This commit moves writing to
`mTerminate` and notifying `mCondition` under the lock to eliminate race
condition and dead lock.

This is not theoretical. It has caused zmu to hang at exit on a
Raspberry Pi 4, exhuasting PHP-FPM process pool. The stacks below are
captured when running ZoneMinder 1.36.12-focal1 on Ubuntu 20.04:

(gdb) thread apply all bt

Thread 2 (Thread 0xffff80c1c880 (LWP 259988)):
#0  futex_wait_cancelable (private=0, expected=0, futex_word=0xaaaae0584e80 <dbQueue+176>) at ../sysdeps/nptl/futex-internal.h:183
#1  __pthread_cond_wait_common (abstime=0x0, clockid=0, mutex=0xaaaae0584e28 <dbQueue+88>, cond=0xaaaae0584e58 <dbQueue+136>) at pthread_cond_wait.c:508
#2  __pthread_cond_wait (cond=0xaaaae0584e58 <dbQueue+136>, mutex=0xaaaae0584e28 <dbQueue+88>) at pthread_cond_wait.c:638
#3  0x0000ffff8700d670 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /lib/aarch64-linux-gnu/libstdc++.so.6
#4  0x0000aaaae0438f08 in zmDbQueue::process (this=0xaaaae0584dd0 <dbQueue>) at ./src/zm_db.cpp:250
#5  0x0000ffff87013fac in ?? () from /lib/aarch64-linux-gnu/libstdc++.so.6
#6  0x0000ffff891264fc in start_thread (arg=0xffffe60d84bf) at pthread_create.c:477
#7  0x0000ffff86dd767c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

Thread 1 (Thread 0xffff80c23010 (LWP 259987)):
#0  __pthread_clockjoin_ex (threadid=281472841926784, thread_return=0x0, clockid=0, abstime=<optimized out>, block=<optimized out>) at pthread_join_common.c:145
#1  0x0000ffff87014240 in std:🧵:join() () from /lib/aarch64-linux-gnu/libstdc++.so.6
#2  0x0000aaaae04314d0 in exit_zmu (exit_code=0) at ./src/zmu.cpp:200
#3  0x0000aaaae042f4c8 in main (argc=<optimized out>, argv=<optimized out>) at ./src/zmu.cpp:797
This commit is contained in:
Ratchanan Srirattanamet 2022-02-20 09:58:03 +00:00
parent fe69261f10
commit ebe502a747
1 changed files with 7 additions and 2 deletions

View File

@ -238,8 +238,13 @@ zmDbQueue::~zmDbQueue() {
}
void zmDbQueue::stop() {
{
std::unique_lock<std::mutex> lock(mMutex);
mTerminate = true;
mCondition.notify_all();
}
if (mThread.joinable()) mThread.join();
}