Software Engineer Intern, Platform Operations

The problem

The platform previously ran every long-running background job on the same VM as the web server. That created three problems: jobs competed for the same pool of resources, a single heavy workload could stall everything else, and enterprise customers with multiple teams on one instance had to coordinate manually to avoid stepping on each other’s compute. Large enough datasets exhausted VM memory entirely, and the operations team would have to resize the machine and pay for that capacity continuously.

The redesign that shipped during my rotation replaced the legacy scheduler with a Hangfire-backed pipeline on Azure Kubernetes Service, using KEDA to scale worker pods in and out based on Redis queue depth. Jobs land in sized queues (Small, Medium, Large) based on predicted resource needs. KEDA watches the queues and spins up pods when work is waiting, scales back to zero when idle. Each job runs in its own container, isolated from everything else.

What I built

I joined Platform Operations when the architecture design was done and owned several backend pieces across approximately 40 merged pull requests.

Metrics and auto-sizing. The dispatcher that routes each job to the right worker size needs historical data on how much CPU and memory past runs actually used. I designed and shipped that persistence layer: an Oracle table and NHibernate-mapped entity for per-job metrics, and a collector service that runs inside every worker pod, samples CPU and memory at a regular interval, and writes a row per minute to the database. The auto-sizing logic reads from this table, estimates the resource profile of the next run, and queues it accordingly.

Getting the schema right took a few iterations. The check constraints I wrote initially were stricter than what real usage produced. Integration testing on production-shaped data surfaced two violations, and I adjusted the constraints once the actual data range became clear. The indexes also needed correcting after I could see the query plan the auto-sizing logic was hitting.

Queue telemetry. To make KEDA queue depth visible in Grafana, I shipped a custom Prometheus gauge that emits the message count per sized Redis queue from the job-housekeeper container. An early version was being emitted from every worker container instead, which double-counted queue depth in dashboards. I caught and fixed the regression before it reached production.

Synthetic load testing. The team needed a way to drive the autoscaler into known states for demos and validation runs. I built a configurable test job with memory cap, duration, and thread count parameters. It ramps memory linearly to the target and burns CPU on the configured threads. It became the standard tool for every autoscaling validation session.

Lifecycle hardening. Once the feature went into integration testing on production-shaped data, edge cases surfaced in the job-state lifecycle: worker pods dying mid-job, the autoscaler scaling in during execution, Redis blips, and a queue-name lookup in the housekeeper that was hardcoded to the wrong queue. That last one caused previously-failed jobs to flip back to a successful state when the housekeeper re-examined them. I traced the root cause across the scheduler, queue state, and database, and wrote fixes for most of the issues in this stream of work.

Removing the old scheduler. Once the new pipeline was the production path for all jobs, I did the full removal of the legacy Quartz-based scheduler and its associated CPU/memory tracker. The refactor touched dozens of files across the monolith and had to leave every existing customer integration path intact.

Results

The feature shipped in a major platform Early Access release. I resolved around 28 user stories and bugs as primary engineer across the four months. Jobs that had previously exhausted the host VM could now run in isolated worker pods that scale on demand and return to zero between runs, and multiple heavy jobs could run in parallel for the first time.