GC Pressure in Spark: How to Find It, Read It, and Actually Fix It
GC pressure is the most misdiagnosed failure in Spark, and the one most often ‘fixed’ by adding heap, which usually makes it worse. Here …
20 min read
Cost Optimization
Field notes from the systems frontier
new
// data engineering · pySpark · orchestration
Notes from the systems frontier
PySpark internals, orchestration, cloud costs, and the lessons learned building reliable data pipelines at scale.
I build robust data pipelines, optimize cloud infrastructure, and specialize in transforming raw data into reliable analytics. This site serves as my digital garden—a place to document field notes, architecture blueprints, and the hard-learned lessons of data engineering.
/
Featured dispatch
Start with the latest
new
deep dive · 20 min read · 4041 words
GC Pressure in Spark: How to Find It, Read It, and Actually Fix It
GC pressure is the most misdiagnosed failure in Spark, and the one most often ‘fixed’ by adding heap, which usually makes it worse. Here is where to look in the UI and the logs, how to tell it apart from data skew, and the levers that actually move the number.
read the postDiscover · Working references
Less browsing. More answers.
Pick the format that matches the moment: scan the commands, review the architecture patterns, or choose the right tool for the pipeline.
2 ARTICLES
SYSTEM DESIGN
Architecture Blueprints
Scalable cloud designs, streaming pipelines, and batch architectures explained.
System Design
Architecture
Distributed Systems
Read blueprints →
0 ARTICLES
Data Modeling
Structuring your warehouse for query performance and long-term maintainability.
Browse all topics →
2 ARTICLES
FINOPS
Cost Optimization
Making complex queries and large clusters cheaper without sacrificing execution speed.
Cost Optimization
FinOps
Cloud Billing
Cut costs →
4 CHEATSHEETS
QUICK REFERENCE
DE Cheatsheet
Short, honest reviews of tools, scripts, and utilities every data engineer needs.
Cheatsheets
Reference
Browse cheatsheets →Welcome to the Field Notes
A short tour of what this site is, the problems I write about, and how to get the most out of the cheatsheets.
2 min read
Site Notes
The Real Math Behind PySpark Resource Sizing
Most Spark tuning guides teach it backwards. Here’s the data-first approach that production engineers actually use, with visual flows and …
7 min read
System Design
keep exploring