Debugging
General
If you use the packaged version of tvheadend, make sure that you use the debug version of tvheadend (with the debugging symbols). For debian/ubuntu these packages have -dbg suffix, for rpm packages, these symbols are in debuginfo rpm files.
If you're going to be regularly trying development versions of Tvheadend or need to report a crash or deadlock then you should really read this page!
If you are investigating problems within Tvheadend then its worth being familiar with tools such as gdb and valgrind or clang, although these are not covered here.
However one thing that can be useful in investigating crashes within Tvheadend is to ensure that coredumps are generated, this will allow post analysis in gdb without having to actual run Tvheadend within gdb.
You can enable temporarily by running:
To make this permanent put this somewhere in your shell environment setup (.bashrc, .profile, etc...) Firstly I'd recommend that if you're specifically trying to investigate an issue then you should consider running Tvheadend manually, rather than as a service, as documented below.
Logging
I'd strongly recommend that if you're specifically trying to investigate a crash or other problem in Tvheadend that you enable debugging:
-s will output debug info to syslog
--debug allows you to specify which subsystem to debug
--trace allows you to enable trace (more in-depth) logging on specific subsystems
You can also get Tvheadend to log to it's own file using:
You may also modify the debug settings using WEB GUI as admin - Configuration/Debugging. Note that the information is not saved, it is just set for run-time (current task).
Debug log path - filename to store log
Debug trace - enable traces
Debug subsystems - comma separated list of subsystems
Trace subsystems - comma separated list of subsystems
The traces must be compiled to the tvheadend binary (see below).
Trace Options
The following options can be passed to tvheadend to provide detailed debugging information while the application is running. Trace debugging has to be enabled at build time with--enable-debugging and can be specified in a CLI command, e.g. tvheadend -u hts -g video --trace <module> or in the web interface (Configuration -> Debugging).
Multiple options can be used, e.g. –-trace cwc,dvr,linuxdvb
Crash reports
Incorrect (not useable) crash reports
In this case, the debug symbols are missing (look to the top of this page). Install the debug version of the tvheadend package.
Correct crash reports
Basic crash debug
You may run tvh in gdb directly using command:
Or attach gdb to the running process:
The 'continue' command will continue the execution of the program. If you need to break the execution and return to gdb, just use 'Ctrl-C'.
You may need to replace tvheadend with the full path to the binary and you will need to replace pid with the PID of the running process. To find that run:
Once you have gdb attached grab a stack trace from every thread using the following command:
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
Enabling coredumps
If you need to investigate some running problem you can always attach (see below) later and if you need to trap crashes, then you can configure your system to generate a core file and then retrospectively analyse this with gdb.
If you're running manually you should enable coredumps in your environment:
I'd recommend you enable this permanently by putting this command in your shell initialisation scripts (.bashrc etc..).
If you're running as a daemon then you should use the -D command line option, this will enable coredumps from the daemon. If you start using sysvinit, upstart etc... then you will need to put this in the configuration file, e.g.:
Finally it's probably worth changing the coredump file format, personally I use the following configuration:
Or put the following in /etc/sysctl.conf:
If you're using a system like Ubuntu that uses apport (and cripples the ability to change the core format) just set core_uses_pid=1 instead.
Note: coredumps are (by default) stored in the current working directory, to make it possible for the daemon to write files the current working directory is set to /tmp when using -D, so check there for core files.
To verify that you have everything configured properly you can use the -A option to force a crash on startup. Do this from the command line or add to /etc/default/tvheadend:
Note: remember to remove the option after you've tested it!
Processing core file.
Once you have a core file you can start up gdb with that coredump, just as if you'd caught the crash while running under gdb:
You may need to replace tvheadend and core above with the proper paths.
For most crashes the most useful information is the back trace, this will provide a stack trace showing where the code crashed and the stack information at the time of the crash:
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
However I'd strongly recommend that you keep a copy of tvheadend binary and core file in case further analysis is required.
Dead or Live Lock
If Tvheadend appears to die but the process is still running, then its quite possible that the process is deadlocked (or possibly live locked).
Buildin deadlock mutex checker (since latest 4.3 version)
Use '--thrdebug 1' as the command line option. The deadlock will be printed to the tvheadend's configuration directory to file mutex-deadlock.txt and to the standard task error output (so you can see it through the systemctl service log for example).
GDB
The best way to help investigate such a problem is to get a full stack trace from every thread in the system.
First attach gdb to the running process:
The 'continue' command will continue the execution of the program. If you need to break the execution and return to gdb, just use 'Ctrl-C'.
You may need to replace tvheadend with the full path to the binary and you will need to replace pid with the PID of the running process. To find that run:
Once you have gdb attached grab a stack trace from every thread using the following command:
Note: "set logging on" will cause GDB to write its output to a file, by default this will be gdb.txt in the current directory.
It might also be useful to generate a core file for good measure:
This information may give an indication as to why things are locked, often 2 threads are stuck trying to lock a mutex (probably each holds the opposite lock).
Reporting crash (or lock)
If you're going to report a crash (or lockup) then please try to provide the above information, including a debug log (or whatever logging you have), a core file and the tvheadend binary and basic information about the platform (distribution, version and architecture) you're running on.
Memory leaks or corruption
It may be really difficult to track these problems. There are basically two tools which may help to discover the memory leaks or memory corruptions.
Valgrind
It is very slow, but it may be useable for things which are triggered everytime:
clang
There is address and leak sanitizer in the clang toolkit.
The clang / llvm tools are usually split to multiple packages, here is list of required packages for Fedora 26:
The binary must be rebuild using the clang compiler and libraries:
Example build script (build_with_clang.sh):
Make sure to make your script executable.
If you do not see resolved the function names like:
get the correct path for the llvm-symbolizer, i.e. with
then make sure that you set the external symbolizer like:
The error log should be like:
Mutex profiling
The code blocks are protected using mutexes. Tvheadend has the debug interface to show locks which took too much time now (latest 4.3 code). Use '--thrdebug 10020' where 20 means 20 millisecond threshold to show the problematic mutexes. The output is printed to standard error file descriptor (stderr) by default. If you run tvheadend using systemctl or initd, you may send those messages to a UDP port. Set the TVHEADEND_RTLOG_UDP_PORT environment variable like:
The result should look like:
Last updated