Modes of Thread debugging in HP WDB

HP WDB offers three modes of Thread debugging:

Interactive Mode
Batch Mode
Attach Mode
The +check Mode

Thread-debugging in Interactive mode

Interactive mode of thread debugging is available for multi-threaded applications running on HP-UX 11iv2, or HP-UX 11iv3.

Using thread-debugging feature in HP WDB

Complete the following steps to use the thread-debugging feature in interactive mode:

Compile the program with –mt option to include threads in compilation:
$ cc –mt –o a.out filename.c
Navigate to the path where gdb is available.
Enter the following command to invoke gdb:
$ ./gdb

Invoke the executable that you want to debug:

(gdb) file <Complete path of the executable or name of the executable>

Enable thread check along with the specific option as required.
(gdb) set thread-check [option][on|off]
Execute the file with the following command:
(gdb) run <Name of the executable>

The thread-check command

The advanced thread debugging features can be enabled only if the set thread-check[on] command is enabled. The following advanced thread debugging options are available for the set thread-check command:

recursive-relock [on|off]
unlock-not-own [on|off]
mixed-sched-policy [on|off]
cv-multiple-mxs [on|off]
cv-wait-no-mx [on|off]
thread-exit-own-mutex [on|off]
thread-exit-no-join-detach [on|off]
stack-util [num]
num-waiters [num]




	NOTE: By default all these options are turned on if you set the command set thread-check on.

Debugging common thread-programming problems

Problem: The thread attempts to acquire a non-recursive mutex that it currently has control.

Consider the following scenario:

Function 1 locks a non-recursive mutex and calls Function 2 without releasing the lock object. If Function 2 also attempts to acquire the same non-recursive mutex, the scenario results in a deadlock. In effect, the program does not proceed with the execution.

Consider the following example enh_thr_mx_relock.c

#include pthread.h
#include string.h
#include stdio.h
#include errno.h

pthread_mutex_t r_mtx;   /* recursive mutex */
pthread_mutex_t n_mtx;   /* normal mutex */
extern void     fatal_error(int err, char *func);

/* Print error information, exit with -1 status. */
void
fatal_error(int err_num, char *function)
{
        char    *err_string;

        err_string = strerror(err_num);
        fprintf(stderr, "%s error: %s\n", function, err_string);
        exit(-1);
}

#define check_error(return_val, msg) {          \
        if (return_val != 0)                            \
                fatal_error(return_val, msg);           \
}

main()
{
        pthread_mutexattr_t     mtx_attr;
        pthread_t               tid1;
        extern void             start_routine(int num);
        int                     ret_val;

        alarm (20);

        /* Initialize the mutex attributes */
        ret_val = pthread_mutexattr_init(&mtx_attr);
        check_error(ret_val, "mutexattr_init failed");

        /* Set the type attribute to recursive */
        ret_val = pthread_mutexattr_settype(&mtx_attr,
                        PTHREAD_MUTEX_RECURSIVE);
        check_error(ret_val, "mutexattr_settype failed");

        /* Initialize the recursive mutex */
        ret_val = pthread_mutex_init(&r_mtx, &mtx_attr);
        check_error(ret_val, "mutex_init failed");

        /* Set the type attribute to normal */
        ret_val = pthread_mutexattr_settype(&mtx_attr,
                        PTHREAD_MUTEX_NORMAL);
        check_error(ret_val, "mutexattr_settype failed");

        /* Initialize the normal mutex */
        ret_val = pthread_mutex_init(&n_mtx, &mtx_attr);
        check_error(ret_val, "mutex_init failed");

        /* Destroy the attributes object */
        ret_val = pthread_mutexattr_destroy(&mtx_attr);
        check_error(ret_val, "mutexattr_destroy failed");

        /* Rest of application code here */

        /*
         * Create a thread
         */
        ret_val = pthread_create(&tid1, (pthread_attr_t *)NULL,
                (void *(*)())start_routine, (void *)1);
        check_error(ret_val, "pthread_create 1 failed");

        /*
         * Wait for the threads to finish
         */
        ret_val = pthread_join(tid1, (void **)NULL);
        check_error(ret_val, "pthread_join: tid1");
}

void
start_routine(int thread_num)
{
        int     ret_val;

        sched_yield();

        /* Lock the recursive lock recursively. */
        ret_val = pthread_mutex_lock(&r_mtx);
        check_error(ret_val, "mutex_lock r_mtx");
        printf("Thread %d - got r_mtx\n", thread_num);

        ret_val = pthread_mutex_lock(&r_mtx);
        check_error(ret_val, "mutex_lock r_mtx");
        printf("Thread %d - got r_mtx\n", thread_num);
  	 		  ret_val = pthread_mutex_unlock(&r_mtx);
        check_error(ret_val, "mutex_unlock r_mtx");
        printf("Thread %d - released r_mtx\n", thread_num);

        ret_val = pthread_mutex_unlock(&r_mtx);
        check_error(ret_val, "mutex_unlock r_mtx");
        printf("Thread %d - released r_mtx\n", thread_num);

        /* Try locking the non-recursive lock recursively */
        ret_val = pthread_mutex_lock(&n_mtx);
        check_error(ret_val, "mutex_lock n_mtx");
        printf("Thread %d - got n_mtx\n", thread_num);

        ret_val = pthread_mutex_lock(&n_mtx);
        check_error(ret_val, "mutex_lock n_mtx");
        printf("Thread %d - got n_mtx\n", thread_num);

        ret_val = pthread_mutex_unlock(&n_mtx);
        check_error(ret_val, "mutex_unlock n_mtx");
        printf("Thread %d - released n_mtx\n", thread_num);

        ret_val = pthread_mutex_unlock(&n_mtx);
        check_error(ret_val, "mutex_unlock n_mtx");
        printf("Thread %d - released n_mtx\n", thread_num);
}

At run-time, the debugger keeps track of each mutex in the application and the thread that currently holds each mutex. When a thread attempts to acquire a lock on a non-recursive mutex, the debugger checks if the thread currently holds the lock object for the mutex.

(gdb) set thread-check recursive-relock on

The debugger transfers the execution control to the user and prints a warning message when this condition is detected.

The following is a segment of the HP WDB output:

Starting program: /home/gdb/enh_thr_mx_relock 
Thread 1 - got r_mtx
Thread 1 - got r_mtx
Thread 1 - released r_mtx
Thread 1 - released r_mtx
Thread 1 - got n_mtx
[Switching to thread 2 (system thread 39774)]
warning: Attempt to recursively acquire non-recursive mutex 2 from thread 2.




	TIP: Release the lock on a non-recursive mutex before attempting to acquire lock on the same object again, to avoid this situation.

Problem: The thread attempts to unlock a mutex or a read-write lock that it does not control.

Consider the following scenario: Thread 1 locks mutex A. Thread 2 unlocks mutex A. This is clearly an attempt from Thread 2 to release the lock on mutex A which was previously locked by Thread 1.

Consider the following example enh_thr_unlock_not_own.c:

#include pthread.h
#include string.h
#include stdio.h
#include errno.h
pthread_mutex_t mtx = PTHREAD_MUTEX_INITIALIZER;

/* Print error information, exit with -1 status. */
void
fatal_error(int err_num, char *function)
{
        char    *err_string;

        err_string = strerror(err_num);
        fprintf(stderr, "%s error: %s\n", function, err_string);
        exit(-1);
}

#define check_error(return_val, msg) {          \
        if (return_val != 0)                            \
                fatal_error(return_val, msg);           \
}

main()
{
        pthread_t               tid1;
        extern void             start_routine(int num);
        int                     ret_val;

        /*
         * Create a thread
         */
        ret_val = pthread_create(&tid1, (pthread_attr_t *)NULL,
                (void *(*)())start_routine, (void *)1);
        check_error(ret_val, "pthread_create 1 failed");

        /*
         * Wait for the threads to finish
         */
        ret_val = pthread_join(tid1, (void **)NULL);
        check_error(ret_val, "pthread_join: tid1");
}

void
start_routine(int thread_num)
{
        int     ret_val;
        ret_val = pthread_mutex_unlock(&mtx);
        check_error(ret_val, "mutex_unlock mtx");
}

This usually is indicative of error in the program logic. Typically, applications are coded to lock and unlock objects on a one-one basis.

The following command enables you to detect this condition in a threaded application.

(gdb) set thread-check unlock_not_own on

The debugger transfers the execution control to the user and prints a warning message when this condition is detected.

The following is a segment of the HP WDB output:

Starting program: /home/gdb/enh_thr_unlock_not_own 
[Switching to thread 2 (system thread 39941)]
warning: Attempt to unlock mutex 1 not owned by thread 2.
0x800003ffeffcc608 in __rtc_pthread_dummy+0 () from ../librtc64.sl




	NOTE: In some rare predictable situations the thread might attempt to unlock an object that it has no control over. For example, an application which instructs the thread to unlock a mutex when it encounters a C++ destructor, irrespective of the history of the processing of the C++ constructor.

Problem: The Thread waits on a mutex or a read-write lock that is held by a thread with a different scheduling policy

Consider the following scenario:

Thread 1 is scheduled using Policy1, SP1. Thread 2 is scheduled using Policy2, SP2. Thread 1 waits for a read-write lock object which is held by Thread 2. Since the scheduling policy of the threads is not the same, there are chances of delay in Thread 2 releasing the lock for the read-write object.

Consider the following example enh_thr_mixed_sched.c:

#include pthread.h
#include errno.h
#include sched.h
#include stdio.h

extern void     *thread1_func(), *thread2_func();
extern void     fatal_error(int err_num, char *func);

pthread_mutex_t         mtx = PTHREAD_MUTEX_INITIALIZER;

/* Print error information, exit with -1 status. */
void
fatal_error(int err_num, char *function)
{
        char    *err_string;

        err_string = strerror(err_num);
        fprintf(stderr, "%s error: %s\n", function, err_string);
        exit(-1);
}

#define check_error(return_val, msg) {                  \
                if (return_val != 0)                    \
                        fatal_error(return_val, msg);   \
        }

void *
thread1_func()
{
        int     ret_val;

        ret_val = pthread_mutex_lock(&mtx);
        check_error(ret_val, "mutex_lock mtx");
        printf("In thread1_func()\n");
        sleep(5);
        ret_val = pthread_mutex_unlock(&mtx);
        check_error(ret_val, "mutex_unlock mtx");

        return((void *)NULL);
}

void *
thread2_func()
{
        int     ret_val;

        ret_val = pthread_mutex_lock(&mtx);
        check_error(ret_val, "mutex_lock mtx");
        printf("In thread2_func()\n");
        sleep(5);
        ret_val = pthread_mutex_unlock(&mtx);
        check_error(ret_val, "mutex_unlock mtx");

        return((void *)NULL);
}

main()
{
        pthread_t       pth_id[2];
        int             ret_val, scope;
        int             old_policy;
        pthread_attr_t  attr;
        struct sched_param      param, old_param;

        /* Initialize the threads attributes object */
        ret_val = pthread_attr_init(&attr);
        check_error(ret_val, "attr_init()");

        /* We want bound threads if they are available. */
        ret_val = pthread_attr_getscope(&attr, &scope);
        check_error(ret_val, "attr_getscope()");
        if (scope != PTHREAD_SCOPE_SYSTEM) {
                scope = PTHREAD_SCOPE_SYSTEM;
                ret_val = pthread_attr_setscope(&attr, scope);
                if ((ret_val != 0) && (ret_val != ENOTSUP))
                        fatal_error(ret_val, "attr_setscope()");
        }

        /* Thread 1 is a high priority SCHED_FIFO thread.*/
        ret_val = pthread_attr_setschedpolicy(&attr, SCHED_FIFO);
        check_error(ret_val, "attr_setschedpolicy() 1");

        param.sched_priority = sched_get_priority_max(SCHED_FIFO);
        ret_val = pthread_attr_setschedparam(&attr, &param);
        check_error(ret_val, "attr_setschedparam() 1");

        ret_val = pthread_create(&pth_id[0], &attr, thread1_func, NULL);
        check_error(ret_val, "pthread_create() 1");

        /* Thread 2 is a low priority SCHED_RR thread. */
        ret_val = pthread_attr_setschedpolicy(&attr, SCHED_RR);
        check_error(ret_val, "attr_setschedpolicy() 2");

        param.sched_priority = sched_get_priority_min(SCHED_RR);
        ret_val = pthread_attr_setschedparam(&attr, &param);
        check_error(ret_val, "attr_setschedparam() 2");

        ret_val = pthread_create(&pth_id[1], &attr, thread2_func, NULL);
        check_error(ret_val, "pthread_create() 2");

        /* Destroy the thread attributes object */
        ret_val = pthread_attr_destroy(&attr);
        check_error(ret_val, "attr_destroy()");

        /* wait for the threads to finish */
        ret_val = pthread_join(pth_id[0], (void **)NULL);
        check_error(ret_val, "pthread_join() 1");

        ret_val = pthread_join(pth_id[1], (void **)NULL);
        check_error(ret_val, "pthread_join() 2");
}

Such a situation does not necessarily result in a deadlock or application errors. However, there might be instances of performance lag issues resulting from the mixed scheduling policies.

The following command enables you to check this condition in a threaded application.

set thread-check mixed-sched-policy[on|off]

The following is a segment of the HP WDB output:

Starting program: /home/gdb/enh_thr_mixed_sched 
In thread1_func()
[Switching to thread 3 (system thread 39724)]
warning: Attempt to synchronize threads 3 and 2 with different scheduling policies.
0x800003ffeffcc608 in __rtc_pthread_dummy+0 () from ../librtc64.sl




	TIP: Consider changing the application such that the threads with the same scheduling policy share the mutex.

Problem: Different threads non-concurrently wait on the same condition variable, but with different associated mutexes.

Consider the following scenario:

Thread 1 with mutex A waiting on conditional variable CV1.

Thread 2 with mutex B waiting on conditional variable CV1.

Consider the following example enh_thr_cv_multiple_mxs.c

#include pthread.h
#include stdlib.h
#include errno.h
pthread_mutex_t job_lock1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t job_lock2 = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t  job_cv = PTHREAD_COND_INITIALIZER;
extern void fatal_error(int err, char *f);

void
producer_thread(pthread_mutex_t* job_lock)
{
        int     ret_val;

        /* Acquire the associated mutex lock */
        if ((ret_val = pthread_mutex_lock(job_lock)) != 0)
          fatal_error(ret_val, "p mtx_lock failed");

        /* Signal the condvar to wakeup one thread */
        if ((ret_val = pthread_cond_signal(&job_cv)) != 0)
          fatal_error(ret_val, "cond_signal failed");

        /* Release the associated mutex */
       if ((ret_val = pthread_mutex_unlock(job_lock)) != 0)
          fatal_error(ret_val, "mtx_unlock failed");
}


void
consumer_thread(pthread_mutex_t* job_lock)
{
        int             ret_val;

        /* Acquire the condvar's associated mutex lock */
        if ((ret_val = pthread_mutex_lock(job_lock)) != 0)
          fatal_error(ret_val, "c mtx_lock failed");

        pthread_cond_wait(&job_cv, job_lock);

        /* Release the associated mutex */
        if ((ret_val = pthread_mutex_unlock(job_lock)) != 0)
          fatal_error(ret_val, "mtx_unlock failed");

}


#define check_error(return_val, msg) {                  \
                if (return_val != 0)                    \
                        fatal_error(return_val, msg);   \
        }

int main(int argc, char* argv[])
{
        pthread_t       tid1, tid2, tid3, tid4;
        pthread_mutex_t *l1, *l2;
        int             ret_val;

        if (argc == 1) {
          fprintf(stderr, "error: no arguments\n");
          exit (1);
        }
        else if (strcmp(argv[1], "bad") == 0) {
          l1 = &job_lock1
          l2 = &job_lock2
        }
        else {
          l1 = l2 = &job_lock1
 }

        alarm(20);

        /* Create two threads to do the work */
        ret_val = pthread_create(&tid1, (pthread_attr_t *)NULL,
                (void *(*)())consumer_thread, (void *) l1);
        check_error(ret_val, "pthread_create 1 failed");

        ret_val = pthread_create(&tid2, (pthread_attr_t *)NULL,
                (void *(*)())producer_thread, (void *) l1);
        check_error(ret_val, "pthread_create 2 failed");

        if (l1 != l2) {
          ret_val = pthread_create(&tid3, (pthread_attr_t *)NULL,
                    (void *(*)())consumer_thread, (void *) l2);
          check_error(ret_val, "pthread_create 1 failed");

          ret_val = pthread_create(&tid4, (pthread_attr_t *)NULL,
                  (void *(*)())producer_thread, (void *) l2);
          check_error(ret_val, "pthread_create 2 failed");
        }
      /* Wait for the threads to finish */
        ret_val = pthread_join(tid1, (void **)NULL);
        check_error(ret_val, "pthread_join: tid1");

        ret_val = pthread_join(tid2, (void **)NULL);
        check_error(ret_val, "pthread_join: tid2");

        if (l1 != l2) {
          ret_val = pthread_join(tid3, (void **)NULL);
          check_error(ret_val, "pthread_join: tid3");

          ret_val = pthread_join(tid4, (void **)NULL);
          check_error(ret_val, "pthread_join: tid4");
        }

        exit(0);
}

void
fatal_error(int err_num, char *function)
{
        char    *err_string;
  err_string = strerror(err_num);
        fprintf(stderr, "%s error: %s\n", function, err_string);
        exit(-1);
}

The following command enables you to check this condition in a threaded application.

(gdb) set thread-check cv-multiple-mxs[on|off]

The debugger transfers the execution control to the user and prints a warning message when this condition is detected.

The following is a segment of the HP WDB output:

Starting program: /home/gdb/enh_thr_cv_multiple_mxs bad
[Switching to thread 4 (system thread 39531)]
warning: Attempt to associate condition variable 0 with mutexes 1 and 2.
0x800003ffeffcc608 in __rtc_pthread_dummy+0 () from ../librtc64.sl

According to pthread implementation, the threads that concurrently wait on a single conditional variable need to specify the same associated mutex.




	TIP: The solution is to correct the application source code in such a way that the condition variable which violates the rule uses the same mutex.

Problem: The thread terminates execution without unlocking the associated mutexes or read-write locks.

Consider the following scenario:

Thread A holds mutex MX. This thread terminates without unlocking the mutex MX. There are other threads in the program that wait to gain control on MX.

The termination of thread A with the locked mutex MXcauses the other threads to be in an endless wait to gain control on MX. This situation is an example of a deadlock where the program execution is dependent on the locked mutex and hence is unable to proceed.

Consider the following example enh_thr_exit_own_mx.c

#include pthread.h
#include stdlib.h
#include errno.h
pthread_mutex_t job_lock1 = PTHREAD_MUTEX_INITIALIZER;
extern void  fatal_error(int err, char *f);

void
producer_thread(pthread_mutex_t* job_lock)
{
        int     ret_val;

        /* Acquire the associated mutex lock */
        if ((ret_val = pthread_mutex_lock(job_lock)) != 0)
          fatal_error(ret_val, "p mtx_lock failed");
}

#define check_error(return_val, msg) {                  \
                if (return_val != 0)                    \
                        fatal_error(return_val, msg);   \
        }

main()
{
        pthread_t       tid;
        int             ret_val;

        /* Create two threads to do the work */
        ret_val = pthread_create(&tid, (pthread_attr_t *)NULL,
        (void *(*)())producer_thread, (void *) &job_lock1);
        check_error(ret_val, "pthread_create 2 failed");

        /* Wait for the threads to finish */
        ret_val = pthread_join(tid, (void **)NULL);
        check_error(ret_val, "pthread_join: tid");

        exit(0);
}

void
fatal_error(int err_num, char *function)
{
        char    *err_string;

        err_string = strerror(err_num);
        fprintf(stderr, "%s error: %s\n", function, err_string);
        exit(-1);
}

The following command enables you to check this condition in a threaded application:

set thread-check thread-exit-own-mutex [on|off]

In such a scenario, the debugger transfers the execution control to the user and displays a warning message.

The following is a segment of the HP WDB output:

Starting program: /home/gdb/enh_thr_exit_own_mx 
[Switching to thread 2 (system thread 39677)]
warning: Attempt to exit thread 2 while holding a mutex 1.
0x800003ffeffcc608 in __rtc_pthread_dummy+0 () from ../librtc64.sl




	TIP: If the remaining segments of the application require access to the locked mutex, modify the code segment of the terminating thread to unlock the mutex before it terminates. If the termination is the result of an exception, then consider using a condition handler (in C++) or POSIX Threads library TRY/FINALLY blocks.

Problem: The thread waits on a condition variable for which the associated mutex is not locked.

Consider the following scenario:

A function has a thread which is associated to the mutex MX. The function calls the POSIX Thread Library routine pthread_cond_wait() () beforeMX is locked.

Consider the following example enh_thr_cv_wait_no_mx.c:

#include pthread.h
#include stdlib.h
#include errno.h

pthread_mutex_t job_lock1 = PTHREAD_MUTEX_INITIALIZER;
pthread_mutex_t job_lock2 = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t  job_cv = PTHREAD_COND_INITIALIZER;
extern void  fatal_error(int err, char *f);

void
producer_thread(pthread_mutex_t* job_lock)
{
        int     ret_val;

        /* Acquire the associated mutex lock */
        if ((ret_val = pthread_mutex_lock(job_lock)) != 0)
          fatal_error(ret_val, "p mtx_lock failed");

        /* Signal the condvar to wakeup one thread */
        if ((ret_val = pthread_cond_signal(&job_cv)) != 0)
          fatal_error(ret_val, "cond_signal failed");

        /* Release the associated mutex */
       if ((ret_val = pthread_mutex_unlock(job_lock)) != 0)
          fatal_error(ret_val, "mtx_unlock failed");
}

void
consumer_thread(pthread_mutex_t* job_lock)
{
        int ret_val;
        pthread_cond_wait(&job_cv, job_lock);
}

#define check_error(return_val, msg) {                  \
                if (return_val != 0)                    \
                        fatal_error(return_val, msg);   \
        }

main()
{
        pthread_t tid1, tid2;
        int ret_val;

        alarm (20);

        /* Create two threads to do the work */
        ret_val = pthread_create(&tid1, (pthread_attr_t *)NULL,
        (void *(*)())consumer_thread, (void *) &job_lock1);
        check_error(ret_val, "pthread_create 1 failed");

        ret_val = pthread_create(&tid2, (pthread_attr_t *)NULL,
        (void *(*)())producer_thread, (void *) &job_lock1);
        check_error(ret_val, "pthread_create 2 failed");

        /* Wait for the threads to finish */
        ret_val = pthread_join(tid1, (void **)NULL);
        check_error(ret_val, "pthread_join: tid1");

        ret_val = pthread_join(tid2, (void **)NULL);
        check_error(ret_val, "pthread_join: tid2");

        exit(0);
}

void
fatal_error(int err_num, char *function)
{
        char    *err_string;

        err_string = strerror(err_num);
        fprintf(stderr, "%s error: %s\n", function, err_string);
        exit(-1);
}

This scenario, where a thread waits on a conditional variable before the associated mutex is locked, is a potential cause of unpredictable results in POSIXlibrary.

The following command in HP WDB enables you to check this condition in a threaded application:

set thread-check cv-wait-no-mx[on|off]

In such a scenario, the debugger transfers the execution control to the user and displays a warning message.

The following is a segment of the HP WDB output:

Starting program: /home/gdb/enh_thr_cv_wait_no_mx 
[Switching to thread 2 (system thread 39559)]
warning: Attempt by thread 2 to wait on condition variable 0 without locking the associated mutex 1.
0x800003ffeffcc608 in __rtc_pthread_dummy+0 () from ../librtc64.sl




	NOTE: This is an additional check that HP WDB provides and is not a POSIX.1 standard requirement for the `pthread_cond_wait()`() routine.

You can determine the function which attempts to wait on the condition, by looking at the batcktrace(bt) of the thread that is reported above.




	TIP: Modify the code segment of the function so it acquires control over the mutex associated to the conditional variable before it calls the routine, `pthread_cond_wait()`().

Problem: The thread terminates execution, and the resources associated with the thread continue to exist in the application because the thread has not been joined or detached.

Consider the following scenario:

Thread A in an application terminates execution successfully or as a result of an exception/cancel. The resources associated with the thread exist in the application until the thread is joined or detached.

Consider the following example enh_thr_exit_no_join_detach.c():

#include pthread.h
#include stdlib.h
#include errno.h

pthread_mutex_t job_lock1 = PTHREAD_MUTEX_INITIALIZER;
extern void fatal_error(int err, char *f);

void
my_thread(void* num)
{
        int ret_val;

        /* Acquire the associated mutex lock */
        if ((ret_val = pthread_mutex_lock(&job_lock1)) != 0)
          fatal_error(ret_val, "p mtx_lock failed");

        printf ("In thread %d\n", (int) num);

        /* Release the associated mutex */
        if ((ret_val = pthread_mutex_unlock(&job_lock1)) != 0)
          fatal_error(ret_val, "mtx_unlock failed");
}

#define check_error(return_val, msg) {                  \
                if (return_val != 0)                    \
                        fatal_error(return_val, msg);   \
        }

main()
{
        pthread_t       tid1, tid2, tid3;
        int             ret_val;

        /* Create two threads to do the work */
        ret_val = pthread_create(&tid1, (pthread_attr_t *)NULL,
                (void *(*)())my_thread, (void *)1);
        check_error(ret_val, "pthread_create 1 failed");

        ret_val = pthread_create(&tid2, (pthread_attr_t *)NULL,
                (void *(*)())my_thread, (void *)2);
        check_error(ret_val, "pthread_create 2 failed");

        ret_val = pthread_create(&tid3, (pthread_attr_t *)NULL,
                (void *(*)())my_thread, (void *)3);
        check_error(ret_val, "pthread_create 3 failed");
   /* Detach thread 1 */
        ret_val = pthread_detach(tid1);
        check_error(ret_val, "pthread_join: tid");

        sleep(5);

        /* Wait for the thread 2 to finishes */
        ret_val = pthread_join(tid2, (void **)NULL);
        check_error(ret_val, "pthread_join: tid");

        exit(0);
}

void
fatal_error(int err_num, char *function)
{
        char    *err_string;

        err_string = strerror(err_num);
        fprintf(stderr, "%s error: %s\n", function, err_string);
        exit(-1);
}

If an application repeatedly created threads without the join or detach operation, it will leak resources that might eventually cause the application to fail.

The following command enables you to check this condition in a threaded application:

set thread-check thread-exit-no-join-detach[on|off]

In such a scenario, the debugger transfers the execution control to the user and displays a warning message.

The following is a segment of the HP WDB output:

Starting program: /home/gdb/enh_thr_exit_no_join_detach 
In thread 1
In thread 2
In thread 3
warning: Attempt to exit thread 4 which has neither been joined nor detached.
0x800003ffeffcc608 in __rtc_pthread_dummy+0 () from ../librtc64.sl




	NOTE: A violation of this condition implies outstanding resources that are not released. If the number of violations is small, or if they occur on an error path that causes abrupt termination of the application, you can disable this check on threads.




	TIP: You can modify the application so that the thread is joined or detached. If a thread is never cancelled, joined, or otherwise passed to threads API, it needs to be detached. The thread can be explicitly joined or detached or created with the detach attribute. If the thread is to be manipulated by a threads API, or the return value of threads is to be considered, then the join operation should be performed to obtain the value and destroy the thread.

Problem: The thread uses more than the specified percentage of the stack allocated to the thread.

Each thread is assigned a specific percentage of the stack when it is created. If the stack allocation is not specified for a thread, the default value is used. The stack allocation cannot be modified after a thread is created.

The application must ensure that the thread stack size is sufficient for all operations of the thread. If a thread attempts to use more space than the allocated stack space, it results in a stack overflow.

Consider the following example:

#include pthread.h
#include stdlib.h
#include stdio.h
#include errno.h
pthread_mutex_t job_lock = PTHREAD_MUTEX_INITIALIZER;
extern void fatal_error(int err, char *f);

void
my_thread()
{
        int     ret_val;
        int more_stack[100];
        static int count = 0;

        sched_yield();

        /* Acquire the associated mutex lock */
        if ((ret_val = pthread_mutex_lock(&job_lock)) != 0)
          fatal_error(ret_val, "p mtx_lock failed");

        for (int i = 0; i < 100; i++)
          more_stack[i] = i;
        for (int i = 0; i < 1000; i++);

        /* Release the associated mutex */
        if ((ret_val = pthread_mutex_unlock(&job_lock)) != 0)
          fatal_error(ret_val, "mtx_unlock failed");

        my_thread();
}


#define check_error(return_val, msg) {                  \
                if (return_val != 0)                    \
                        fatal_error(return_val, msg);   \
        }

main()
{
        pthread_t       tid;
        int             ret_val;

        /* Create two threads to do the work */
        ret_val = pthread_create(&tid, (pthread_attr_t *)NULL,
                (void *(*)())my_thread, (void *) NULL);
  check_error(ret_val, "pthread_create 2 failed");

        /* Wait for the threads to finish */
        ret_val = pthread_join(tid, (void **)NULL);
        check_error(ret_val, "pthread_join: tid");

        exit(0);
}

void
fatal_error(int err_num, char *function)
{
        char    *err_string;

        err_string = strerror(err_num);
        fprintf(stderr, "%s error: %s\n", function, err_string);
        exit(-1);
}

The set thread-check stack-util[num] command checks if any thread has used more than the specified percentage[num] of the stack allocation.

The debugger transfers the execution control to the user and displays a warning message when this condition is detected.

The following is a segment of the HP WDB output:

(gdb) set thread-check stack-util 101
Invalid value: stack utilization must be between 0 and 100.
(gdb) set thread-check stack-util 80
(gdb) run
Starting program: /home/gdb/enh_thr_stack_util 
[Switching to thread 2 (system thread 39877)]
warning: Thread 2 exceeded stack utilization threshold of 80%.
0x800003ffeffcc608 in __rtc_pthread_dummy+0 () from ../librtc64.sl

This warning indicates that the thread attempts to exceed its stack utilization limit. This may cause memory access violations, bus errors, or segmentation faults, if the stack utilization reaches 100%.




	TIP: Increase the stack allocation available to the thread when you create it Change the code running in the thread to reduce its use of stack space

Problem: The number of threads waiting on any pthread object exceeds the specified threshold number.

This check identifies contention that results from too many threads attempting to acquire the same lock object.

The set thread-check num-waiters [num] command checks if the number of threads waiting on any pthread object exceeds the specified threshold number [num].

The debugger transfers the execution control to the user and displays a warning message when this condition is detected.

A relatively large number of threads waiting on pthread synchronization object can indicate a performance constraint on the application.




	TIP: To avoid this condition: Check if any of the data object shared among the application threads can be accessed using its own synchronization object. Check if the program has too many threads whose activity depends on concurrent access to the contended mutex.

Thread-debugging in Batch mode

HP WDB 5.8 supports batch mode of debugging threads for HP-UX 11iv2 and later, on Integrity systems and on HP-UX 11i v3 in PA-RISC systems for 64 bit applications.

The debugger provides a log file with the list of thread-related errors that occur in the application.

In batch mode, the debugger detects the all the thread-conditions that are detected during an interactive debugging session.

The debugger reports extended information such as variable address, name, id and other specifications related to the involved pthread objects. In addition, it displays the stack trace of the executing thread at the point of error.

Pre-requisites for Batch Mode of Thread Debugging

The various prerequisites for Batch mode of Thread Debugging are as follows:

The thread-debugging feature in HP WDB is dependent on the availability of the dynamic linker B.11.19 or later versions.
Advanced thread-debugging feature requires the pthread tracer library which is available by default on systems running HP-UX 11i v2 or later.

Steps to debug threads in Batch Mode

Compile the source files.
Set the LD_LIBRARY_PATH environment variable, based on the platform as follows:
- For IPF 32 bit applications, set
  export LD_LIBRARY_PATH=/opt/langtools/wdb/lib/hpux32
- For IPF 64 bit applications, set
  export LD_LIBRARY_PATH=/opt/langtools/wdb/lib/hpux64
- For PA 64 bit applications, set
  export LD_LIBRARY_PATH=/opt/langtools/wdb/lib/pa20_64
Map the share libraries as private for HP 9000 systems using the following command:
$ chatr +dbg enable ./executable
NOTE: This step is not applicable for Integrity systems.

Create a configuration file, rtcconfig to specify the various thread conditions that you want the debugger to detect.

NOTE: The configuration file contains lines of the following form:

set thread-check [on|off] | [option] [on|off] | [option] [num]

And/Or

set frame-count [num]

And/Or

files = <name of the executable on which the thread checking is to be done>

Set the environment variable BATCH_RTC to on as export set BATCH_RTC=on

Complete one of the following steps to preload the librtc runtime library:

Set the target application to preload librtc by using the +rtc option for the chatr command. In addition to automatically loading the librtc library, the +rtc option for the chatr command also maps the shared libraries as private. To enable or disable the target application to preload the librtc runtime library, enter the following command at the HP-UX prompt:

$ chatr +rtc <enable|disable> <executable>




	NOTE: The `chatr +rtc` option preloads the `librtc` runtime library from the following default paths: For 32 bit IPF applications, `/opt/langtools/lib/hpux32/librtc.so` For 64 bit IPF applications, `/opt/langtools/lib/hpux64/librtc.so` For 64-bit PA applications, `/opt/langtools/lib/pa20_64/librtc.sl`

To preload the librtc runtime library from a path that is different from the default paths, you must use the LD_PRELOAD environment variable.

Instead of automatically preloading librtc and mapping the shared libraries, you can explicitly preload the required librtc library after mapping the shared libraries private.
In the case of HP 9000 systems, you must explicitly map the share libraries as private by using the +dbg enable option for the chatr command, as follows:
$ chatr +dbg enable ./<executable>
(This step is not required on Integrity systems.)
To explicitly preload the librtc runtime library and start the target application, enter one of the following commands:
- For 32 bit IPF applications,
  LD_PRELOAD=/opt/langtools/lib/hpux32/librtc.so <executable>
- For 64 bit IPF applications,
  LD_PRELOAD=/opt/langtools/lib/hpux64/librtc.so <executable>
- For 64-bit PA applications,
  LD_PRELOAD=/opt/langtools/lib/pa20_64/librtc.sl <executable>
If LD_PRELOAD and chatr +rtc are used to preload the librtc runtime library, the librtc runtime library is loaded from the path specified by LD_PRELOAD.

If HP WDB detects any thread error condition during the application run, the error log is output to a file in the current working directory.

The output file has the following naming convention:

<executablename>.<pid>.threads

where pid is the process id.

Limitations in Batch Mode of Thread Debugging

The feature does not obtain the thread-error information in batch mode for forked process in a multiprocessing application. However, if the librtc.sl library is preloaded, the debugger obtains the thread-error information in the batch mode for exec-ed application.

You cannot specify an alternate output directory for the thread-error log. The thread-error log file is output into the current working directory only.

HP WDB cannot execute both batch mode thread check and batch mode heap check together. If the rtcconfig file has both entries, then batch heap check overrides the batch thread check.

Known issues with thread debugging for interactive and batch mode

Issue 1:

During the execution of advanced thread checking for applications that fork, in the interactive mode, the following message appears if the GDB follows the child:

Pthread analysis file missing!

This error message appears because the thread-error information for the forked process is not available.

However, if the forked process exec()s another binary, the thread-error information is available for the exec -ed binary.

Issue 2:

In both interactive and batch modes, if the applications exceed their thread stack utilization, the following error message appears:

Error accessing memory address

This occurs when GDB attempts a command line call on an already overflowing thread stack.

Thread- debugging in Attach Mode

HP WDB provides support to attach a running process to the debugger. To use thread debugging commands after attaching GDB to a running process, complete the following steps:

Set LD_LIBRARY_PATH to include the appropriate directory, by entering one of the following commands:
- For 32 bit IPF applications,
  export LD_LIBRARY_PATH=/opt/langtools/wdb/lib/hpux32
- For 64 bit IPF applications,
  export LD_LIBRARY_PATH=/opt/langtools/wdb/lib/hpux64
- For 32 bit PA applications,
  export LD_LIBRARY_PATH=/opt/langtools/wdb/lib
- For 64-bit PA applications,
  export LD_LIBRARY_PATH=/opt/langtools/wdb/lib/pa20_64

Complete one of the following steps to preload the librtc runtime library:

Set the target application to preload librtc by using the +rtc option for the chatr command. In addition to automatically loading the librtc library, the +rtc option for the chatr command also maps the shared libraries as private.

To enable or disable the target application to preload the librtc runtime library, enter the following command at the HP-UX prompt:

$ chatr +rtc <enable|disable> <executable>




	NOTE: The `chatr +rtc` option preloads the `librtc` runtime library from the following default paths: For 32-bit IPF applications, `/opt/langtools/lib/hpux32/librtc.so` For 64-bit IPF applications, `/opt/langtools/lib/hpux64/librtc.so` For 32-bit PA applications, `opt/langtools/lib/librtc.sl` For 64-bit PA applications, `/opt/langtools/lib/pa20_64/librtc.sl`

To preload the librtc runtime library from a path that is different from the default paths, you must use the LD_PRELOAD environment variable.

Instead of automatically preloading librtc and mapping the shared libraries, you can explicitly preload the required librtc library after mapping the shared libraries private.
In the case of HP 9000 systems, you must explicitly map the share libraries as private by using the +dbg enable option for the chatr command, as follows:
$ chatr +dbg enable ./<executable>
(This step is not required on Integrity systems.)
To explicitly preload the librtc runtime library and start the target application, enter one of the following commands:
- For 32-bit IPF applications,
  LD_PRELOAD=/opt/langtools/lib/hpux32/librtc.so <executable>
- For 64-bit IPF applications,
  LD_PRELOAD=/opt/langtools/lib/hpux64/librtc.so <executable>
- For 32-bit PA applications,
  LD_PRELOAD=/opt/langtools/lib/librtc.sl <executable>
- For 64-bit PA applications,
  LD_PRELOAD=/opt/langtools/lib/pa20_64/librtc.sl <executable>

If LD_PRELOAD and chatr +rtc are used to preload the librtc runtime library, the librtc runtime library is loaded from the path specified by LD_PRELOAD.

Complete one of the following steps:
- Attach the debugger to the required process and enable thread debugging, as follows:
  gdb -thread -p <pid>
  or
  gdb -thread <executable> <pid>
- Alternately, you can attach the process to the debugger and consequently invoke thread debugging, as follows:
  $ gdb <executable> <pid> ... (gdb)set thread-check on

Thread Debugging in +check Mode

The +check=thread compiler option enables batch mode thread debugging features of HP WDB.




	NOTE: This feature is available only for compiler versions A.06.20 and later.

It is a convenient way of launching the batch mode advanced thread checking features without setting any other environment variables at runtime. In other words, batch mode thread checking has two modes of invocation. The first method is to use the run- time environment variables LD_LIBRARY_PATH, LD_PRELOAD and BATCH_RTC on existing precompiled applications. The second method is to use the +check=thread option at the compile time.

+check=thread must only be used with multithreaded programs. It is not enabled by +check=all. This functionality requires HP WDB 5.9 or later.

The default configuration used by +check=thread option is as follows:

thread-check=1;recursive-relock=1;unlock-not-own=1;
mix-sched-policy=1;cv-multiple-mxs=1;cv-wait-no-mx=1;
thread-exit-own-mutex=1;thread-exit-no-join-detach=1;stack-util=80;
num-waiters=0;frame_count=4;output_dir=.;

Behavior of the +check=thread option can be changed by users by providing their own rtcconfig file. The user specified rtcconfig file can be in the current directory or in a directory specified by the GDBRTC CONFIG environment variable.

If any thread error condition is detected during the application run, the error log will be output to a file in the current working directory. The output file will have the following naming convention:

<executable name>.<pid>.threads,

where, <pid> is the process identifier.

Modes of Thread debugging in HP WDB

Technical documentation

» Table of Contents

» Index

Thread-debugging in Interactive mode

Using thread-debugging feature in HP WDB

The thread-check command

Debugging common thread-programming problems

Problem: The thread attempts to acquire a non-recursive mutex that it currently has control.

Problem: The thread attempts to unlock a mutex or a read-write lock that it does not control.

Problem: The Thread waits on a mutex or a read-write lock that is held by a thread with a different scheduling policy

Problem: Different threads non-concurrently wait on the same condition variable, but with different associated mutexes.

Problem: The thread terminates execution without unlocking the associated mutexes or read-write locks.

Problem: The thread waits on a condition variable for which the associated mutex is not locked.

Problem: The thread terminates execution, and the resources associated with the thread continue to exist in the application because the thread has not been joined or detached.

Problem: The thread uses more than the specified percentage of the stack allocated to the thread.

Problem: The number of threads waiting on any pthread object exceeds the specified threshold number.

Thread-debugging in Batch mode

Pre-requisites for Batch Mode of Thread Debugging

Steps to debug threads in Batch Mode

Limitations in Batch Mode of Thread Debugging

Known issues with thread debugging for interactive and batch mode

Thread- debugging in Attach Mode

Thread Debugging in +check Mode