[TEARDOWN] YYYY-MM-DD WIP

Paper Title: Subtitle if Applicable

Full citation: Author(s), "Paper Title," Conference/Journal, Year. arXiv:XXXX.XXXXX

LLM

Teardown by: Author Name, Inference Foundry Core Team

Profiling hardware: NVIDIA H100 SXM5 80GB · CUDA 12.4 · PyTorch 2.4.0+cu124

Abstract

REPLACE WITH ABSTRACT TEXT.

Background

Notation

Throughout this article the following notation is used consistently:

\( N \) — sequence length (number of tokens)
\( d_{\text{model}} \) — model hidden dimension
\( d_k \) — attention key/query projection dimension
\( d_v \) — attention value projection dimension
\( H \) — number of attention heads
\( B \) — batch size

Prerequisites

REPLACE WITH PREREQUISITES TEXT.

Mathematical Deconstruction

3.1 Core Formulation

REPLACE WITH TEXT. Example structure below — delete before submitting.

The scaled dot-product attention mechanism is defined as:

\[ \text{Attention}(Q, K, V) = \softmax\!\left(\frac{QK^\top}{\sqrt{d_k}}\right)V \quad \in \R^{N \times d_v} \tag{1} \]

where \( Q \in \R^{N \times d_k} \), \( K \in \R^{N \times d_k} \), \( V \in \R^{N \times d_v} \) are the query, key, and value matrices respectively.

3.2 Complexity Analysis

REPLACE WITH COMPLEXITY ANALYSIS.

The naive attention algorithm requires:

Compute: \( O(N^2 d_k + N^2 d_v) \) floating-point operations
Memory: \( O(N^2) \) for the attention weight matrix \( S = QK^\top / \sqrt{d_k} \)

3.3 Key Algorithmic Changes

REPLACE WITH ALGORITHMIC ANALYSIS.

3.4 Numerical Stability Considerations

REPLACE WITH NUMERICAL ANALYSIS.

Hardware Profiling

Profiling Environment

GPU Model:          NVIDIA H100 SXM5 80GB
GPU Firmware:       96.00.74.00.01
CUDA Version:       12.4
cuDNN Version:      9.1.0
Driver Version:     550.54.15
OS:                 Ubuntu 22.04.4 LTS
Kernel:             6.5.0-28-generic
CPU:                Intel Xeon Platinum 8480+
RAM:                2048 GiB DDR5 ECC
PyTorch Version:    2.4.0+cu124
Triton Version:     3.0.0

Full environment spec and raw CSV data: data/PAPER-SLUG/env-HARDWARE-SLUG.txt

4.1 VRAM Consumption

REPLACE WITH VRAM ANALYSIS. Include the table below populated with real data.

Peak VRAM consumption (MiB) — measured with `torch.cuda.max_memory_allocated()`
Batch Size	Seq Len 256	Seq Len 1024	Seq Len 4096	Seq Len 8192
1	XXXX	XXXX	XXXX	XXXX
8	XXXX	XXXX	XXXX	XXXX
32	XXXX	XXXX	XXXX	XXXX
128	XXXX	XXXX	XXXX	OOM

4.2 Time-to-First-Token (TTFT)

REPLACE WITH TTFT ANALYSIS. 10 warmup runs discarded; median of 100 timed runs reported.

Time-to-First-Token (ms) — median of 100 runs, 10 warmup runs discarded
Batch Size	Seq Len 256	Seq Len 1024	Seq Len 4096	Seq Len 8192
1	XX.X	XX.X	XXX.X	XXXX.X
8	XX.X	XX.X	XXX.X	XXXX.X
32	XX.X	XX.X	XXX.X	XXXX.X
128	XX.X	XX.X	XXX.X	OOM

4.3 Decode Throughput

REPLACE WITH THROUGHPUT DATA.

4.4 Kernel-Level Profiling

REPLACE WITH KERNEL PROFILING DATA.

Nsight Compute metrics — seq_len=4096, batch_size=1, BF16
Metric	Measured Value	Peak Theoretical	Utilization (%)
SM Occupancy	XX warps/SM	64 warps/SM (H100)	XX%
HBM Bandwidth	X.XX TB/s	3.35 TB/s (H100 SXM5)	XX%
FLOP/s (BF16 Tensor)	X.XX PFLOP/s	989 TFLOP/s (H100 SXM5)	XX%

4.5 CGo / FFI Bridging Overhead

REPLACE WITH CGo PROFILING DATA, or delete section if not applicable.

$ go test -bench=BenchmarkCGoCall -benchmem -count=10 ./...
BenchmarkCGoCall-64    1000000    1143 ns/op    0 B/op    0 allocs/op

Raw benchmark data and pprof flame graph: data/PAPER-SLUG/cgo-bench.txt

Architectural Bottlenecks

5.1 Roofline Analysis

REPLACE WITH ROOFLINE ANALYSIS.

The arithmetic intensity of the attention GEMM is computed as: \[ I = \frac{\text{FLOP}}{\text{bytes transferred}} = \frac{2 \cdot N^2 \cdot d_k}{(2 N d_k + 2 N d_k + 4 N^2) \cdot \text{sizeof}(\text{dtype})} \]

Roofline plot (generated from ncu data, script at data/PAPER-SLUG/plot.py):

Roofline plot for PAPER kernel on H100 SXM5 — Roofline model — PAPER kernel, H100 SXM5, BF16. Measured point shown in red.

5.2 Identified Bottlenecks

REPLACE WITH BOTTLENECK ANALYSIS.

Bottleneck classification by operating regime
Regime	Class	Limiting Resource	Evidence
Short seq (N ≤ 512)	LATENCY-BOUND	Kernel launch overhead	nsys timeline: kernel gap > kernel runtime
Long seq (N ≥ 4096)	MEMORY-BOUND	HBM bandwidth	I < ridge point; HBM util = XX%

5.3 Optimization Hypotheses

REPLACE WITH OPTIMIZATION HYPOTHESES.

Reproduction Notes

6.1 Environment Setup

# Clone the repository and install dependencies
git clone https://github.com/Inference-Foundry/Crucible.git
cd Crucible

# Install Python dependencies (all standard library or widely-used packages only)
pip install torch==2.4.0+cu124 --index-url https://download.pytorch.org/whl/cu124
pip install pandas matplotlib

6.2 Running the Profiling Suite

# VRAM and throughput sweep
python data/PAPER-SLUG/profile_vram.py --device cuda --dtype bf16

# Nsight Compute kernel profiling (requires sudo or nsight compute permissions)
ncu --metrics sm__warps_active.avg.pct_of_peak_sustained_active,\
dram__bytes.sum,l1tex__t_bytes.sum \
--export data/PAPER-SLUG/ncu-report \
python data/PAPER-SLUG/run_kernel.py

6.3 Known Failure Modes

REPLACE WITH KNOWN ISSUES.

References

A. Author, B. Author, "Paper Title," in Proceedings of the Conference Name (CONFNAME), City, Country, Year, pp. XX–XX. [Online]. Available: arXiv:XXXX.XXXXX