Understanding legacy C code is a difficult task because
it is linked together tightly. New feature addition to this legacy code is even
more tedious than writing the feature independently from scratch. Majority of
the teams in VMware deals with some form of legacy code. There is often a
dilemma on how to validate and enhance this legacy code. One approach to
solving this is refactoring the code with unit tests. In this paper, we
describe a 3-layered approach to the Unit test framework where in each layer we
are addressing a unique problem.
The bottom layer, is where we design a set of generic
steps to build and compile your legacy C code with Google test framework
binaries and mock out all the dependent modules which might get invoked while
running these unit tests. The middle layer, is where we have extended the
framework to explain how we can run these Unit tests as part of continuous
integration and deployment (CI/CD) pipeline. Product components are assembled
by Jenkins CI/CD pipeline where they are built, unit tested, run through some
lightweight functional verification tests. This is integral for continuous
delivery in cloud development model. The top layer, is where we also generate
code coverage results as part of these Unit tests. This helps identify unused
or stale legacy code in the product for cleanup and areas lacking any coverage
can be addressed by writing more unit tests.
With VMware as a company shifting towards cloud
delivery model service, unit tests are essential for faster delivery and
quickly narrowing down issues. This paper not just helps in making writing Unit
tests simpler but also helps fill up functional test gaps of a legacy code.
The IEEE definition of unit
testing is the testing of individual hardware or software units or groups of
related unit 1,xiii. Writing unit tests for a legacy code is tedious and
labor intensive. While we have strong evidences, which proves that unit testing
can be leveraged to improve product quality, it is still not widely used as an
integral part of programming specially when it comes to legacy code which
mostly isn’t written with unit test in mind. It is a classic example of chicken
and egg dilemma: To write unit tests I need to refactor code and to refactor
code safely I need unit tests.
In this paper, we present an approach
to writing unit tests for Plain Old C code. The real difficulty in testing C
code is breaking the dependencies on external modules so you can isolate code
in units 2,ix. This can be especially
problematic when you are trying to get tests around legacy code. We have tons
of unit test frameworks available in the market for various languages but
integration of any one of them for Legacy C code was not directly applicable.
The rest of the paper is organized as follows: First, we
explain the Motivation behind the Idea to stress on the challenges we faced.
Second, we provide the details of our current solution and the work we have
done so far. Last but not the least we lay out the Future work that we have
planned for FoCUT.
2.1 Current Framework Limitations
Even though multiple unit test
frameworks already exist for C code, they lack the ability to isolate the code
by removing internal or external dependencies to make it unit testable. Some
which do provide isolation are mostly tedious and time-consuming and are often
complicated to implement.
C codes are linked together
tightly and it is harder to break dependencies during run time. Think of
virtual functions in Object Oriented Programming which serves this purpose.
Because of this tight coupling of C, there are basically two tools available
which are being leveraged for isolating the function under test – the C
preprocessor and the linker.
In the first approach, we can
eliminate functions either by using #define or by surrounding them using
compiler directives such as #if TEST … #endif to prevent them being compiled
into the released code. But this
requires modification in the production code which is not always desirable for
stable legacy codes. Using the linker substitution method, the code can easily
be replaced by mocks or stubs during linking time, the only problem being
rewriting of extra wrapper function for each mock or stub.
2.2 Test Driven Development
Test Driven Development (TDD) is a software development
practice which relies on the repetition of small development cycles. These
include writing of automated unit tests, which initially fail, for new feature
addition or improvement and then writing the minimum amount of code to pass
those tests and finally refactoring the code. TDD requires more code to be
written due to the inclusion of more number of unit tests which might seem to
be time consuming.
TDD results into more modularized and flexible code. It
requires the code to be treated and written as independent isolated units, tested
independently and then integrated together. These large number of independent
tests help in limiting the number of defects in the code. This isolation can
only be achieved efficiently by use of mock objects and stubs, which also helps
in the modularization of code. Present C unit test frameworks are not all
compliant with this TDD model, which makes TDD much more complex and
2.3 Code Gaps
Code coverage is a measure of the code covered under tests,
which includes the number of lines, blocks, arcs of code executed while running
the automated tests. Though having high number of code coverage does not
necessarily mean the system is completely tested but it does provide a good
metric of how much testing is being done. The biggest challenge in inheriting a
Legacy code is to understand the flow and identify the current test coverage
that we have for it. A lot of times there might be huge chunks of stale code
which is never been called or used. Functional tests cannot always cover all
parts of the code in a way in which unit tests can. All conditions and branches
can only be satisfied if proper mocking functionality is there in case of
3. The 3 layered Approach: FoCUT
Framework for C Unit Test (FoCUT) is our solution to
support most of the features one would expect from any standard unit testing
framework. The framework bundles all the different features such as testing,
mocking, generation of code coverage and integration of tests with test
pipelines into a single test framework.
The framework uses the open source Google Test
framework which provides standard automated testing functionalities such as
text fixtures or passing test states. Test fixtures provides the functionality
of setting up and destroying the functions that can be shared with multiple
test cases. Google Test provides C++ libraries for writing unit tests which can
be leveraged for writing unit tests for C source codes.
Fig. 1. Sample Test Code
To be modified
3.1 Bottom layer
In FoCUT we support the mocking functionality using the
GNU linker features to substitute the desired function call with the intended
or fake one during the linking stage. It uses the concept of wrapper functions
which wraps the original function call and implements the desired one. The mocks
are desired behavior of the real function which is specified by an individual
test case. At present the generating a mock function is done manually, but this
framework aims to automate this process. For writing a mock an extra flag is
added during the linking of test codes with the source codes which is
-Wl,–wrap,functionName where, functionName is the name of the function which
needs to be substituted with the desired one. The desired implementation of the
function is added using the __wrap symbol with original function name, eg.,
__wrap_functionName(). Using this the desired behavior of the function can be
defined. For having multiple implementations of the same module this concept is
combined with the concept of function pointers.
Fig. 2. Sample Mock Code
To be modified
3.2 Middle Layer
The tool also provides the feature to integrate the
test results with Continuous Integration and Continuous Development (CI/CD)
pipelines such as Jenkins. We automate all the process required for TDD: automation
of unit tests to automation of the generation of mocks. These have been the
major challenge w.r.t other available C related unit test frameworks. FoCUT
will not just make writing unit test less laborious but will also reduce the
time we spend in writing these tests with this automation.
3.3 Top Layer
The framework also supports the feature for generation
of file-based code coverage taking the advantage of already existing Micro Unit
Test (UUT) Framework’s utrun tool 3, vii. Code quality is directly dependent
on the quality of unit test. More the code is covered, less is the chance of
breaking anything in the system. Code coverage helps identify which parts of
legacy code is already by existing tests so that more tests can be added to the
uncovered code 4,v.
FoCUT generates code coverage using the GNU’s gcov
tool. The framework provides the file based code coverage percentage for each
source file under test. At present the tool, provides line based coverage but
can be easily extended to provide branch based as well.
Likewise, now known as
PowerBroker Identity Services is an open source software that centralizes
authentication for Unix, Linux and Mac environments by extending Microsoft’s
Active Directory’s Kerberos authentication and single sign-on capabilities to
these platforms 5,xii.
VMware maintains an internal copy
of this open source software and leverages it to support the following use
Have a centralized configuration management by
ensuring all ESXi servers and vCenters, deployed in Customer’s environment, are
part of their Active Directory domain.
Allow users to leverage their AD credentials to
access ESXi and vCenters in their environment.
Attain consistent configuration by extending
native group policy management tools to include settings 6,xii for ESXi and
Consolidate directories to simplify management
of complex Customer environments.
We have used Likewise as the
legacy C code to develop FoCUT. The following work has been successfully done
so far for Likewise:
There are three basic building blocks for the
likewise code: Likewise Security and Authentication Sub System (lsassd),
Netlogond and Likewise I/O Manager (lwiod). Using FoCUT we have so far covered
the major code flows for all these three services and have written close to 150
unit tests for the same. We have also added unit tests for the Domain join
module as well since it is one of the important use case of Likewise. Since likewise
modules are written in plain old C code, we have used the implementation
details described under Section 3.3 of this paper to write the unit tests.
UUT Framework is a C/C++ framework for
facilitating Micro (?) Unit Tests through code extraction and a “walled
garden” approach 7,vii. Utrun is
the tool which is used to run local unit tests. We have written a new plugin
‘likewise-gtest’ in UUT Framework’s ‘utrun’ tool. This plugin automates the
process of building the likewise code, execution of the unit tests and the generation
of the result for likewise unit tests. This reduces the manual effort of doing
all the above tasks individually. The plugin also generates a consolidated
Jenkins pipeline consumable XML report from the generated results of the unit
tests run. Thus, this single plugin ‘likewise-gtest’, which is now available as
part of FoCUT, facilitates the integration of C unit tests with build CI/CD pipelines.
We further extended the ‘likewise-gtest’ plugin
in FoCUT, to add file-based code coverage generation process. The plugin now generates
the line based code coverage for each test suite which is mapped to each file
for which tests have been written. The code coverage results are now shown as
part of the unit test run final output in the Jenkins CI/CD pipeline.
As part of writing unit tests for legacy code we
also discovered a bug in the existing code where we were doing a memcpy (memory
copy) instead of memcmp (memory compare). FoCUT kept flagging this function call as
failing until we went and modified the code.
5. Future Work
We have presented a simple but effective approach to
writing unit tests for legacy C code. Most testing methods do not check
behavioral results, but focus only on defining what to test. Often many tests
are required to cover all cases, manually checking test results severely
hampers its effectiveness, and makes repeated and frequent testing impractical
8,v. To remedy this, we aim to automatically generate unit test mockups from
formal specifications. In the current approach, the programmer still must
supply actual test data by hand. In the future, we hope to partially alleviate
this problem by automatically generating some of test inputs from the
Another area of future work is to gain more experience
with our approach. The application of our approach so far has been limited to
successfully writing ~150 unit tests for one of the products which our team
owns. We have also started on using FoCUT for other components as well of the
Platform Service Controller stack. We soon aim to extend our framework to other
teams as well to leverage and use it for their products. This will help give a
more generic outlook to the framework and it will no longer be limited to
vSphere as a product.