Automated Performance Timeline Comparison

In this work, we propose an approach to localize software performance regressions through an automated call tree comparison scheme. We have implemented this in a tool called Zam, which takes call trees generated through Chrome DevTools (called "timelines") as input, and suggests a hierarchically arranged set of function nodes to inspect.

Journal Paper (STVR'21): Localizing software performance regressions in web applications by comparing execution timelines (Preprint)

Motivation

Software performance regression localization is an important task, as it leads software developers to potential fixes for performance issues introduced by code changes. For web applications, one commonly used approach is to compare performance timelines - i.e., call trees annotated with runtime performance data - and extract the lowest-level regressing functions. This process is very expensive, not only because of the large size of the call tree, but also because of inherent non-determinism (from our perspective as humans) introduced by JavaScript's asynchrony and event-driven execution. We therefore need to find a more efficient way to perform these timeline comparisons.

Methodology

To address the issue stated above, we propose a design that takes multiple pairs of execution timelines as input and performs a heuristic-based, pairwise comparison of the timelines. The approach performs the comparison on a per-level basis in the call tree, and it uses a hybrid node matching algorithm to determine which nodes correspond to each other across two different timelines. We implemented the design in a tool called Zam, and our evaluation of the tools points to its accuracy, real-world applicability, and usefulness in an industrial setting.

Tool

Zam: The tool itself is closed-source, but experimental artifacts (e.g., timelines, aggregated results) will be made available.