Performance Bottleneck in Quarto and Litedown Resolved After Discovery of Hidden Quadratic Complexity in R Connections

The global data science community recently witnessed a significant breakthrough in the performance optimization of Quarto, the next-generation open-source scientific and technical publishing system. Earlier this month, a performance regression was identified that rendered Quarto nearly 100 times slower than its predecessor, R Markdown, when handling documents with extensive output. The discovery, triggered by a GitHub issue filed by a user identified as @idavydov, led to a deep dive into the memory management of the R programming language, ultimately revealing a long-standing efficiency trap within the R base function textConnection().

The Catalyst: GitHub Issue 14156 and the 100x Performance Gap

The investigation began when @idavydov filed issue #14156 against the quarto-cli repository. The report highlighted a striking disparity in rendering times. For a document containing a single code chunk designed to generate 100,000 lines of text—using the R command cat(strrep("xn", 100000))—Quarto took approximately 35 seconds to complete the rendering process. In contrast, the older rmarkdown::render() function finished the same task in less than half a second.

This performance delta was not merely a minor inconvenience; it represented a critical scalability issue for researchers and data scientists who rely on Quarto for large-scale simulations, logs, and data-heavy reporting. The reporter also noted that the same bottleneck appeared to plague litedown, a newer, lightweight alternative to knitr developed by Yihui Xie, the creator of knitr and a prominent figure in the R ecosystem.

Chronology of the Investigation: From Symptom to Root Cause

Yihui Xie, who maintains litedown independently of the Quarto and knitr core systems, took up the challenge to investigate the discrepancy. Because litedown utilizes xfun::record() to execute R code and capture output, Xie focused his initial efforts there.

The diagnostic process followed a rigorous technical timeline:

1. Initial Reproduction: Xie successfully replicated the 100-fold slowdown within the litedown environment, confirming that the issue was not specific to Quarto’s command-line interface but resided in how output was being handled at the R level.
2. Standard Profiling: Using R’s built-in tool utils::Rprof(), Xie analyzed the execution of xfun::record(). The preliminary results indicated that the cat() function was responsible for 88% of the total runtime. The call stack showed a path through one_string and paste, leading Xie to believe that the overhead was caused by string concatenation prior to writing the output.
3. Garbage Collection Analysis: Suspecting there was more to the story, Xie re-profiled the code, this time enabling garbage collection profiling with the argument gc.profiling = TRUE. This proved to be the turning point in the investigation.
4. The Revelation: The new profile revealed that 56% of the total runtime was occupied by the Garbage Collector (GC). This shifted the diagnosis from a "slow function" problem to a "memory churn" problem. The system was not struggling to process data; it was struggling to clean up an astronomical number of short-lived objects created during the execution.

Technical Analysis: The O(n^2) Trap in textConnection

The core of the problem was located within the handle_output() function of xfun::record(). To capture the output of an R code chunk, the system was using sink() in conjunction with a textConnection.

The implementation relied on the following logic:
con = textConnection('out', 'w', local = TRUE)

In R, a writable textConnection is designed to append elements to a character vector, one for each line of output. When the command cat(strrep("xn", 100000)) is executed, R enters a tight loop to process 100,000 lines. Because R vectors are "copy-on-modify" objects, every time a new line is added to the out vector, R does not simply append it to the end of the existing memory block. Instead, it allocates an entirely new, larger vector, copies all existing elements from the old vector into the new one, and then adds the new line.

Mathematically, this represents a quadratic complexity, or $O(n^2)$. For 100,000 lines, the system performs approximately five billion operations related to copying data ($1 + 2 + 3 + … + 100,000$). Every intermediate vector created during this process is immediately discarded, creating a massive workload for the Garbage Collector.

Comparative Benchmarks: Why R Markdown Remained Fast

A critical question for the development team was why the legacy R Markdown system did not suffer from this same slowdown. R Markdown utilizes the knitr package, which in turn uses the evaluate package for code execution.

The investigation found that the evaluate package handles output capture differently. Instead of using textConnection(), which builds a character vector in R’s high-level memory, it sinks output into a file() connection. This approach delegates buffering and writing to the operating system’s file I/O layer. Because the operating system handles byte-stream writing efficiently without the copy-on-modify overhead of R character vectors, it sidesteps the quadratic growth trap entirely.

The Engineering Solution: Transitioning to rawConnection

To resolve the issue in litedown and Quarto without introducing the overhead of physical file system interaction, Xie implemented a fix using rawConnection().

Unlike textConnection(), which manages a vector of strings, rawConnection() utilizes a dynamically growing byte buffer. This buffer follows a growth pattern similar to the realloc() function in the C programming language, typically doubling in size when it reaches capacity. This change reduces the complexity of appending data from $O(n^2)$ to an amortized $O(1)$.

The implementation change was concise but impactful. The system now captures output as a raw byte stream, which is converted to a single character string and then split into lines only once, at the very end of the process.

Before Fix (The Slow Path):

• Create textConnection.
• Sink output (each line triggers a copy-on-modify event).
• Resulting vector out is already populated but at great computational cost.

After Fix (The Fast Path):

• Create rawConnection with raw(0).
• Sink output into the byte buffer.
• Retrieve the final buffer using rawConnectionValue().
• Convert bytes to characters using rawToChar().
• Split the string by newlines in a single pass.

Quantitative Impact and Results

Following the implementation of the rawConnection fix in the xfun package, the performance gains were immediate and substantial. In a controlled test using 50,000 lines of output:

• Original Runtime: 5.58 seconds.
• Post-Fix Runtime: 1.30 seconds.
• Efficiency Gain: Approximately 4.3x for 50,000 lines.

When scaled to the 100,000 lines reported in the initial GitHub issue, the performance improvement becomes even more dramatic due to the elimination of the quadratic growth curve. The "cat" and "GC" spikes previously seen in profiling data effectively vanished, confirming that the memory management bottleneck had been neutralized.

Broader Implications for the R Ecosystem

This incident serves as a significant case study for software developers working within high-level languages like R. It highlights the distinction between "idiomatic" code and "efficient" code.

The textConnection() function is a standard, documented part of base R and is even used within other base functions like capture.output(). For the vast majority of use cases—such as capturing a few dozen or even a few hundred lines of output—the performance penalty is negligible and invisible to the user. However, as data science tasks grow in scale, these "hidden" $O(n^2)$ traps can become catastrophic failure points.

The resolution of this issue is expected to have several long-term effects:

1. Quarto Stability: Quarto users will see improved reliability when generating large reports, particularly in automated environments where log output can be voluminous.
2. Profiling Best Practices: The R community has been reminded of the critical importance of gc.profiling = TRUE. Without accounting for garbage collection, developers may spend weeks optimizing the wrong functions while the true culprit remains hidden in memory management.
3. Base R Scrutiny: This event may prompt further review of other legacy base R functions that rely on character vector accumulation, potentially leading to more widespread use of raw byte buffers for high-performance I/O.

As Quarto continues to position itself as the industry standard for reproducible research, the rapid identification and resolution of this performance bottleneck by the open-source community demonstrates the robustness of the ecosystem. The fix has been integrated into the development versions of the underlying R packages, ensuring that the next release of Quarto will provide the high-speed performance users have come to expect.