Enabling gcov code coverage

Aaron Arvey aarvey at cs.hmc.edu
Tue Nov 9 18:14:01 CST 2004


We're using gcov to measure how well the Wine test suite
covers the wine source tree.  Here's our second cut at
making it easy to run wine compiled for coverage testing.

In order to apply and use this patch, you'll need to do something like:
      $> patch -p0 < enablegcov.patch
      $> autoconf
      $> ./configure --enable-gcov

Changelog:
  * configure.ac
    new option "--enable-gcov"
  * Make.rules.in
  * loader/Makefile.in
    new variable LDEXTRACFLAGS (for loader EXTRACFLAGS)
    used instead of EXTRAFLAGS in the loader/Makefile.in
    (low level compilation issues when profiling code)
  * documentation/windev-otherdebug.sgml
    Added in documentation for using gcov on wine
Index: Make.rules.in
===================================================================
RCS file: /home/wine/wine/Make.rules.in,v
retrieving revision 1.179
diff -u -u -r1.179 Make.rules.in
--- Make.rules.in	7 Oct 2004 03:12:44 -0000	1.179
+++ Make.rules.in	9 Nov 2004 23:31:13 -0000
@@ -104,7 +104,8 @@
 api_manext      = 3w
 conf_manext     = 5
 CLEAN_FILES     = *.o *.a *.so *.ln *.$(LIBEXT) \\\#*\\\# *~ *% .\\\#*
*.bak *.orig *.rej \
-                  *.flc *.spec.c *.spec.def *.dbg.c *.tab.c *.tab.h
@LEX_OUTPUT_ROOT at .c core
+                  *.flc *.spec.c *.spec.def *.dbg.c *.tab.c *.tab.h
@LEX_OUTPUT_ROOT at .c core \
+                  *.bb *.bbg *.da

 OBJS = $(C_SRCS:.c=.o) $(EXTRA_OBJS)

Index: configure.ac
===================================================================
RCS file: /home/wine/wine/configure.ac,v
retrieving revision 1.325
diff -u -u -r1.325 configure.ac
--- configure.ac	9 Nov 2004 20:16:35 -0000	1.325
+++ configure.ac	9 Nov 2004 23:31:14 -0000
@@ -16,6 +16,7 @@
 AC_ARG_ENABLE(debug, AC_HELP_STRING([--disable-debug],[compile out all
debugging messages]))
 AC_ARG_ENABLE(trace, AC_HELP_STRING([--disable-trace],[compile out TRACE
messages]))
 AC_ARG_ENABLE(win64, AC_HELP_STRING([--enable-win64], [build a Win64
emulator on AMD64 (won't run Win32 binaries)]))
+AC_ARG_ENABLE(gcov, AC_HELP_STRING([--enable-gcov],[turn on code coverage
analysis tools]))

 AC_ARG_WITH(opengl,    AC_HELP_STRING([--without-opengl],[do not use
OpenGL]))
 AC_ARG_WITH(curses,    AC_HELP_STRING([--without-curses],[do not use
curses]))
@@ -811,6 +812,7 @@
 dnl **** Check for gcc specific options ****

 AC_SUBST(EXTRACFLAGS,"")
+AC_SUBST(LOADEREXTRACFLAGS,"")
 if test "x${GCC}" = "xyes"
 then
   EXTRACFLAGS="-Wall -pipe"
@@ -872,6 +874,31 @@
     EXTRACFLAGS="$EXTRACFLAGS -gstabs+"
   fi

+  dnl Check for --enable-gcov and add appropriate flags for gcc
+  dnl Note that these extra switches are NOT applied to the loader
+  if test "x$enable_gcov" = "xyes";
+  then
+    dnl Check for -fprofile-arcs and -ftest-coverage option
+    AC_CACHE_CHECK([for gcc -fprofile-arcs support],
+		    ac_cv_c_gcc_fprofile_arcs,
+                    [WINE_TRY_CFLAGS([-fprofile-arcs],
+		                     ac_cv_c_gcc_fprofile_arcs="yes",
+				     ac_cv_c_gcc_fprofile_arcs="no")])
+    AC_CACHE_CHECK([for gcc -ftest-coverage support],
+		    ac_cv_c_gcc_ftest_coverage,
+                    [WINE_TRY_CFLAGS([-ftest-coverage],
+		                     ac_cv_c_gcc_ftest_coverage="yes",
+				     ac_cv_c_gcc_ftest_coverage="no")])
+    if test "$ac_cv_c_gcc_fprofile_arcs" = "yes" && \
+       test "$ac_cv_c_gcc_ftest_coverage" = "yes"
+    then
+      EXTRACFLAGS="$EXTRACFLAGS -fprofile-arcs -ftest-coverage"
+      dnl Turn off optimization so code coverage tool
+      dnl can get accurate line numbers
+      EXTRACFLAGS=`echo "$EXTRACFLAGS" | sed -e 's/-O[0-9]*//g'`
+    fi
+  fi
+
   dnl Check for noisy string.h
   saved_CFLAGS="$CFLAGS"
   CFLAGS="$CFLAGS -Wpointer-arith -Werror"
@@ -885,6 +912,11 @@
   fi
 fi

+dnl When adding in more 'EXTRAFLAGS' you must put them above this!
+dnl Strips out the code profiling abilities because the preloader
+dnl relies on low level assumptions which added code messes up.
+LOADEREXTRACFLAGS=`echo "$EXTRACFLAGS" | sed -e 's/-fprofile-arcs
-ftest-coverage//g'`
+
 dnl **** Check how to define a function in assembly code ****

 AC_CACHE_CHECK([how to define a function in assembly code],
ac_cv_asm_func_def,
Index: documentation/winedev-otherdebug.sgml
===================================================================
RCS file: /home/wine/wine/documentation/winedev-otherdebug.sgml,v
retrieving revision 1.1
diff -u -u -r1.1 winedev-otherdebug.sgml
--- documentation/winedev-otherdebug.sgml	26 Oct 2004 22:45:47 -0000
1.1
+++ documentation/winedev-otherdebug.sgml	9 Nov 2004 23:31:14 -0000
@@ -497,9 +497,177 @@
 	</listitem>
       </orderedlist>
     </sect1>
+    <sect1>
+      <title>Which code is not yet tested?</title>
+      <para>
+        The Wine test suite doesn't actually test all of Wine.
+	Surprised?  You shouldn't be.  It's difficult to write a truly
+	complete test suite.  Fortunately, gcc comes with a tool
+	called 'gcov' that will tell you exactly which code is not yet
+	tested.
+      </para>
+      <para>
+        To use gcov on wine, do the following:
+      </para>
+      <orderedlist>
+        <listitem>
+         <para>
+           Configure wine with the <literal>--enable-gcov</literal>
+           option to <command>configure</command>, then do a clean
+           build of wine as normal.
+         </para>
+       </listitem>
+        <listitem>
+         <para>
+           Run part or all of the Wine test suite.  For example, if
+	   you're interested in how well
+	   <filename>kernel32.dll</filename> is tested, just do
+	   <screen>
+cd dlls/kernel/tests
+rm *.ok
+make test
+           </screen>
+         </para>
+       </listitem>
+        <listitem>
+         <para>
+           Run gcov on the source files for the part of Wine you're
+           interested in testing.  For instance,
+	   <screen>
+cd dlls/kernel
+for a in *.c; do gcov $a; done
+           </screen>
+	   For each file, it will output statistics about what
+           percentage of the code has been tested, as well as an
+           annotated source file showing exactly which lines were
+           exercised by the test suite.
+         </para>
+       </listitem>
+      </orderedlist>
+      <para>
+        For instance, to find out what lines of
+        <filename>dlls/lzexpand/lzexpand_main.c</filename> are not yet
+        tested by <filename>dlls/lzexpand/tests</filename>, you might
+        do (the following example uses an out-of-tree build):
+        <screen>
+cvs checkout wine
+mkdir build
+cd build
+../wine/configure --enable-gcov
+make depend && make
+cd dlls/lzexpand/tests
+make test
+cd ..
+gcov ../../../wine/dlls/lzexpand/lzexpand_main.c
+  0.00% of 3 lines executed in file ../../../wine/include/wine/unicode.h
+  Creating unicode.h.gcov.
+  0.00% of 4 lines executed in file /usr/include/ctype.h
+  Creating ctype.h.gcov.
+  0.00% of 6 lines executed in file /usr/include/bits/string2.h
+  Creating string2.h.gcov.
+  100.00% of 3 lines executed in file ../../../wine/include/winbase.h
+  Creating winbase.h.gcov.
+  50.83% of 240 lines executed in file
../../../wine/dlls/lzexpand/lzexpand_main.c
+  Creating lzexpand_main.c.gcov.
+less lzexpand_main.c.gcov
+        </screen>
+        Look through the output file
+	<filename>lzexpand_main.c.gcov</filename> for a line that has
+	<literal>####</literal> instead of a line count.  Here's one,
+	in the function <function>LZOpenFile</function>:
+        <screen>
+        9:  545:        if ((mode&~0x70)!=OF_READ)
+        6:  546:                return fd;
+        3:  547:        if (fd==HFILE_ERROR)
+    #####:  548:                return HFILE_ERROR;
+        3:  549:        cfd=LZInit(fd);
+        3:  550:        if ((INT)cfd <= 0) return fd;
+        3:  551:        return cfd;
+        </screen>
+        <command>gcov</command> output consists of three components:
+        the number of times a line was run, the line number, and the
+        actual text of the line.  (If a line is optimized out by the
+        compiler, it will appear as if it was never run, so
+        <literal>--enable-gcov</literal> disables optimization.)  gcov
+        is telling us that line 548 is not exercised by the test
+        suite.  This is bad, because it means the test suite might
+        miss a bug.  In this case, fixing the problem is simple: since
+        this is the only place <function>LZOpenFile</function> returns
+        <constant>HFILE_ERROR</constant>, we just have to add a
+        situation to
+        <filename>dlls/lzexpand/tests/lzexpand_main.c</filename> where
+        <function>LZOpenFile</function> ought to return
+        <constant>HFILE_ERROR</constant>, and verify that it does.
+        For example, the following lines added to
+        <filename>dlls/lzexpand/tests/lzexpand_main.c</filename> would
+        do the job:
+        <screen>
+INT file;

+/* Check for non-existent file. */
+file = LZOpenFile("badfilename_", &amp;test, OF_READ);
+ok(file == LZERROR_BADINHANDLE,
+   "LZOpenFile succeeded on nonexistent file\n");
+LZClose(file);
+       </screen>
+       Once we add this, we naturally want to find out if the new line
+       actually does exercise the line in question.  To do this, clear
+       the histograms (line counts) by removing
<filename>*.da</filename>, remove
+       the old <filename>*.gcov</filename> output files, remove the
+       <filename>*.ok</filename> so <literal>make test</literal> knows
+       it needs to actually run the test, and rerun it:
+      </para>
+      <screen>
+rm *.da *.gcov
+cd tests
+rm *.ok
+make
+make test
+cd ..
+gcov ../../../wine/dlls/lzexpand/lzexpand_main.c
+  0.00% of 3 lines executed in file ../../../wine/include/wine/unicode.h
+  Creating unicode.h.gcov.
+  0.00% of 4 lines executed in file /usr/include/ctype.h
+  Creating ctype.h.gcov.
+  0.00% of 6 lines executed in file /usr/include/bits/string2.h
+  Creating string2.h.gcov.
+  100.00% of 3 lines executed in file ../../../wine/include/winbase.h
+  Creating winbase.h.gcov.
+  51.67% of 240 lines executed in file
../../../wine/dlls/lzexpand/lzexpand_main.c
+  Creating lzexpand_main.c.gcov.
+less lzexpand_main.c.gcov
+      </screen>
+      <para>
+         Sure enough, gcov reported that
+	 <filename>lzexpand_main.c</filename> has 51.67% coverage
+	 instead of 50.83%.  That's nearly a one percent improvement,
+	 which is awfully good.  The output file confirms this:
+      </para>
+      <screen>
+       10:  545:        if ((mode&~0x70)!=OF_READ)
+        6:  546:                return fd;
+        4:  547:        if (fd==HFILE_ERROR)
+        1:  548:                return HFILE_ERROR;
+        3:  549:        cfd=LZInit(fd);
+        3:  550:        if ((INT)cfd <= 0) return fd;
+        3:  551:        return cfd;
+      </screen>
+      <para>
+        So there you have it: that's how to use gcov to increase the
coverage
+	of the wine test suite.
+      </para>
+      <para>
+        For a further in depth description of gcov, see the official
+        gnu gcov documentation at <ulink
+        url="http://gcc.gnu.org/onlinedocs/gcc-3.2.3/gcc/Gcov.html">
+        http://gcc.gnu.org/onlinedocs/gcc-3.2.3/gcc/Gcov.html</ulink>,
+        or the excellent article written by Steve Best for Linux
+        Magazine, at <ulink
+        url="http://www.linux-mag.com/2003-07/compile_01.html">
+        http://www.linux-mag.com/2003-07/compile_01.html</ulink>.
+      </para>
+    </sect1>
   </chapter>
-
 <!-- Keep this comment at the end of the file
 Local variables:
 mode: sgml
Index: loader/Makefile.in
===================================================================
RCS file: /home/wine/wine/loader/Makefile.in,v
retrieving revision 1.17
diff -u -u -r1.17 Makefile.in
--- loader/Makefile.in	11 Aug 2004 20:59:09 -0000	1.17
+++ loader/Makefile.in	9 Nov 2004 23:31:14 -0000
@@ -23,6 +23,10 @@

 LIBPTHREAD  = @LIBPTHREAD@
 LDEXECFLAGS = @LDEXECFLAGS@
+LOADEREXTRACFLAGS = @LOADEREXTRACFLAGS@
+
+.c.o:
+	$(CC) -c $(INCLUDES) $(DEFS) $(DLLFLAGS) $(LOADEREXTRACFLAGS)
$(CPPFLAGS) $(CFLAGS) -o $@ $<

 wine-glibc: glibc.o Makefile.in
 	$(CC) -o $@ glibc.o $(LIBWINE) $(LIBPORT) $(LIBPTHREAD)
$(EXTRALIBS) $(LDFLAGS)




More information about the wine-patches mailing list