B.Tech CSE (AI&ML) '27 · Open to roles
LLM TRAINING SYSTEMS · AI AGENTS · MECHANISTIC INTERPRETABILITY

AKSHAT BALYAN

I train language models from scratch on consumer hardware and build ▍. From 60M-parameter architecture ablations and 1 Mbp-context DNA models to safety layers for parallel coding agents and real-time computer vision in production.

0
60M Models Trained From Scratch
0+ hrs
Controlled GPU Training
0 Mbp
DNA Context In 5.6GB VRAM
<0ms
CV Multi-Stream Latency
Research & Publications
[01 / 05]
Preprint // Mechanistic Interpretability

Architectural Inductive Biases Under Extreme Capacity Constraints: A Mechanistic Case Study of 60M-Parameter Language Models

Faculty-advised team project. Trained twenty-four ~60M-parameter GPT-style models on 1B FineWeb-Edu tokens (250+ GPU hours) under a single-axis protocol comparing MHA/MQA/GQA/MLA, RoPE, MoE, and Muon vs. AdamW — isolating why mechanisms designed for 7B–70B models break when compressed.

  • MLA Rank Bound (Prop. 3): MLA's deficit at r=64 (PPL 33.84) is rank saturation. Widening to r=256 recovered -1.94 PPL, and a decoupled-RoPE branch (M29) reached 31.22 PPL at 2.7× lower KV cache than MHA.
  • Muon on GQA (Prop. 1): Plain Muon on GQA was a null (32.10 → 32.09). Shape-aware update scaling with retuned lr=0.01 (M28) unlocked 31.08 PPL (-1.01 PPL).
  • KDA Finite-State Recall Bound (Prop. 2): Linear attention trades exact recall for fixed-size state; LAMBADA recall collapsed to 0.116 while BLiMP grammar stayed top-tier at 0.782.
  • Perplexity ≠ Capability: Shared-expert MoE models (M14–M16) lead on downstream benchmarks (PIQA 0.608, ARC-Easy 0.485) despite worse perplexity.
PyTorch Muon Optimizer MLA Kimi Delta Attention MoE Routing SVD Spectra
Paper // Genomic Language Models

Mnemos-DNA: Long-Context DNA Language Model with K-mer Fast-Weight Memory

Faculty-advised team project. Linear-attention (KDA) models scale to long DNA sequences but lose exact recall — the same finite-state bound found in the 60M study. Mnemos adds a k-mer fast-weight memory to restore it.

  • Exact recall restored to 64 kbp for only ~3% extra parameters.
  • 1 Mbp-context inference in 5.6 GB VRAM on an 8 GB laptop GPU.
  • Released weights on Hugging Face and a companion benchmark, recall-bench, on PyPI.
PyTorch Linear Attention (KDA) Fast-Weight Memory Hugging Face PyPI
Systems & Architecture Engineering
[02 / 05]
Aura — Agentic SRE Co-pilot AI Agents

Detects incidents, links them to recent deploys, recalls similar past incidents via Qdrant + Neo4j memory, and proposes fixes like rollbacks and restarts — with human approval required for anything destructive. Ships a terminal UI, web dashboard, and failure-injection demo; runs on Docker Compose or Kubernetes.

CodeMind-1B Pipeline Tech Report

Memory-efficient PyTorch pipeline (gradient checkpointing, chunked cross-entropy, hybrid Muon + AdamW) fusing MLA and MoE (~1.5B total / ~400M active) with Multi-Token Prediction. Trains a 1.5B model on an 8GB RTX 5060 at 1,940 tok/s; full run on a $10 compute budget.

HackRx 6.0 Multi-Modal RAG Top 35 / 10,000+

RAG API that answers questions over textbook- and legal-scale PDFs on the fly, without pre-computed embeddings. Combines AWS Titan and Gemini with token budgeting to keep latency low. Containerized and deployed on AWS App Runner with CI/CD through GitHub Actions.

WhatsApp AI Agent (MCP) ★ 11 · 4 Forks

Autonomous WhatsApp agent built on Google ADK and the Model Context Protocol: a Go messaging bridge plus a Python MCP server exposing tools for multi-step tasks.

Experiment Results, Explained
[03 / 05]

Results from the 60M-parameter study above. Every model is identical except for one change (attention type, optimizer, etc.), so any difference in score comes from that change alone. Scores are mostly perplexity (PPL) — how surprised the model is by unseen text; lower is better. Pick a tab to see one experiment.

Experience & Education
[04 / 05]
Software Engineer Intern
Oct 2025 – Sep 2026
AMRR Techsols // Real-Time Computer Vision & Backend
  • Built and deployed real-time computer vision systems for factory label-legibility and burner inspection, processing 3 synchronized camera streams at 30–60 FPS with <100ms inference latency.
  • Optimized on-site inference pipelines to process continuous video streams without dropping frames.
  • Built backend services in FastAPI and Django for a client-facing platform, turning client requirements into modular, low-latency APIs.
B.Tech, Computer Science & Engineering (AI & ML)
2023 – 2027
Ajay Kumar Garg Engineering College, Ghaziabad
Technical Matrix & Honors
[05 / 05]
AI / ML PyTorch, Hugging Face, LLM training & fine-tuning, RAG, AI agents (LangGraph, MCP, Google ADK), MoE, MLA, RoPE, Muon, YOLOv8
Backend & Cloud FastAPI, Django, REST APIs, Docker, Kubernetes, AWS (App Runner, ECR, EC2), GitHub Actions
Data PostgreSQL, SQLite, Qdrant, Neo4j, Pinecone, ChromaDB, PyArrow
Languages Python, Go, SQL, JavaScript / TypeScript
Honors • Top 35 Finalist — Bajaj Finserv HackRx 6.0 (10,000+ teams)
• 3-Star on CodeChef
• NVIDIA DLI: Fundamentals of Deep Learning