I help engineering teams cut their Spark and Databricks costs and build data platforms that scale.

What I do

Spark Performance Optimization — I analyze your physical query plans (not just your code) to find the anti-patterns that are actually costing you money: missed broadcast joins, unnecessary shuffles, scan inefficiencies, and join strategy problems. Every finding comes with an estimated savings so you know what to fix first.

Data Pipeline Architecture — Full platform assessments for teams running Spark, Kafka, Airflow, and Databricks. I deliver architecture documents with diagrams, prioritized optimization roadmaps, and cost reduction projections you can present to leadership.

Event-Driven Systems Design — Kafka architecture, streaming pipeline design, and event-driven platform modernization. I’ve built production event-driven systems at Riot Games (gaming analytics), StockX (marketplace), and May Mobility (autonomous vehicles).

Background

I’m a Staff Data Engineer with 15 years of experience building data platforms at Riot Games, Microsoft, Amazon, StockX, May Mobility, and Aspiration. I hold a BSE from the University of Waterloo and a Stanford NLP/Deep Learning certificate.

Selected highlights:

  • 99%+ cost reduction on production PySpark workloads at Riot Games
  • 10x throughput optimization on real-time pricing systems at StockX
  • End-to-end real-time analytics pipeline (Kafka to Databricks) at May Mobility
  • Avro Schema Registry pilot for high-traffic Kafka clusters at StockX
  • Cluster Yield — my Spark performance analytics platform (WIP SaaS)

Engagement options

I offer consulting through Fiverr for scoped engagements, or direct for longer-term work.

Spark Plan Audit — Send me your physical plan, get back a written audit with savings estimates. 3-day turnaround.

Performance Optimization — Full audit plus implemented fixes, refactored code, config recommendations, and a 30-minute walkthrough call. 7-day turnaround.

Architecture Consulting — Full platform review with architecture document, optimization roadmap, cost projections, and two video calls. 14-day turnaround.

For custom engagements or longer-term consulting, reach out directly.

Contact

Email: travisneilturner@gmail.com
LinkedIn: linkedin.com/in/neil-turner-8978115