Nazmus Sakib

নাজমুস সাকিব

Lecturer, Computer Science and EngineeringBRAC University

I study how AI agents fail, and how we can tell when they do. My research spans AI for Software Engineering and SRE, LLM post-training, and AI safety. I work at the point where language models stop being demos and start being systems people depend on.

I am a Lecturer in Computer Science and Engineering at BRAC University and hold a BSc from BUET. I am applying to PhD programs for Fall 2027 and welcome research collaborations.

Research

AI for Software Engineering and SRE
LLM Post-Training
AI Safety
NLP
Agentic Systems
Human-Computer Interaction

Passing Without Repairing: Verifier Blind Spots in a Live SRE Agent Benchmark

Nazmus Sakib

Accepted NeurIPS 2026 workshop “Who Verifies the Agents?”, poster

With the Service deleted and no agent acting, the benchmark's grader still reports the fault as mitigated, because it only checks Deployment and Pod health.

Paper Code
Seen but Not Bound: LLM Operations Agents Treat Confirmation Policies as a Caution Dial, Not a Gate

Nazmus Sakib, Md Rizwan Parvez

Under review

Agents can tell a production target from staging, yet a confirmation rule meant for production shifts their behavior nearly as much everywhere else.

Code
FaithfulCut

Nazmus Sakib

Under review ICSE 2027 Industry Challenge Track

Replayed with only the results a deployed selector could see, a leading history-based test selector let nearly three times as many failing builds slip through, while selectors that read the change barely noticed.

Code
Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?

Nazmus Sakib

Preprint arXiv:2609.24144

Giving every rollout in a GRPO group the same luck lifted success under tool faults from 61.8% to 66.8%, but not under grader noise.

arXiv Code
MemAgent: A Cache-Inspired Framework for Augmenting Conversational Web Agents With Task-Specific Information

Nazmus Sakib, Protoy Barai, Sifat Ishmam Parisa, Anindya Iqbal

Published WOA 2025, CEUR Workshop Proceedings Vol. 4028

MemAgent asks for task details once and caches them with an expiry date, so recurring web tasks took 58% less time in a 15-person study.

Paper Code

All research

Open source

SREGym

Live benchmark for AI SRE agents
- Merged
  
  Reported that the alert check can fail a correct fix while the fault's own alerts are still clearing, so faster fixes failed more often, and opened a fix that waits for alerts to resolve.
  
  Issue #1116 Pull request #1117
- Merged
  
  Added a Kafka poison-pill problem: one malformed record stalls a consumer while every pod still looks healthy.
  
  Pull request #790
ITBench

Open-source benchmark for IT automation
- Merged
  
  Reported problems in five OpenTelemetry demo scenarios and fixed three: a network policy fault that blocked nothing, an alert that fired on healthy idle services, and a fault aimed at a Deployment that does not exist.
  
  Issue #1339 Pull request #1340
- Merged
  
  Reported and fixed an OpenTelemetry demo install that timed out after a chart update renamed its span metrics connector.
  
  Issue #1323 Pull request #1324
AIOpsLab

Framework for evaluating autonomous AIOps agents
- In review
  
  Reported that the mitigation check passes when the broken workload is deleted or scaled to zero, and opened a fix that also catches unschedulable pods.
  
  Issue #208 Pull request #209
Kestra
- Merged
  
  Reported and fixed custom tool schemas that did not reach the Anthropic API intact, and added a blueprint that labels support tickets in Postgres with Claude Message Batches.
  
  Issue #147 Pull request #148 Pull request #979
Hugging Face
- Released
  
  Models and datasets, including a clinical chain-of-thought GPT-OSS-20B in GGUF with over 4,000 downloads, and GeoQL-4B for text to OverpassQL.
  
  gpt-oss-20b-clinical-cot-gguf GeoQL-4B-SFT

News

Oct 2026
Opened two pull requests to Kestra: a tool-schema fix for the Anthropic plugin and a Claude Message Batches blueprint.
Oct 2026
Seen but Not Bound is under review, with code public in OPharm.
Sep 2026
Passing Without Repairing was accepted as a poster at the NeurIPS 2026 workshop “Who Verifies the Agents?”
Sep 2026
The SREGym blog featured the benchmark problems our UIUC++ SRSE cohort built from real postmortems, including mine.
Jul 2026
Posted Luck Is Not Skill on arXiv.
May 2026
My Kafka poison-pill problem was merged into SREGym during the UIUC++ SRSE research internship.
Sep 2025
MemAgent was published in the WOA 2025 proceedings (CEUR, Vol. 4028).
Mar 2025
Graduated from BUET with a BSc in Computer Science and Engineering.

Research

Publications & interests

I care about what happens after the benchmark number: how agents remember, how post-training actually behaves, and how people work alongside these systems.

2026Peer-reviewed workshop paper

Passing Without Repairing: Verifier Blind Spots in a Live SRE Agent Benchmark

With the Service deleted and no agent acting, the benchmark's grader still reports the fault as mitigated, because it only checks Deployment and Pod health.

Accepted NeurIPS 2026 workshop “Who Verifies the Agents?”, poster

AI for SREBenchmark verifiersMutation testingSREGymKubernetesPre-registered

Paper Code

2026

Seen but Not Bound: LLM Operations Agents Treat Confirmation Policies as a Caution Dial, Not a Gate

Agents can tell a production target from staging, yet a confirmation rule meant for production shifts their behavior nearly as much everywhere else.

Under review

Code

2026

FaithfulCut

Replayed with only the results a deployed selector could see, a leading history-based test selector let nearly three times as many failing builds slip through, while selectors that read the change barely noticed.

Under review ICSE 2027 Industry Challenge Track

Software testingContinuous integrationTest selectionEvaluation methodologyLLMsPre-registered

Code

2026

Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?

Giving every rollout in a GRPO group the same luck lifted success under tool faults from 61.8% to 66.8%, but not under grader noise.

Preprint arXiv:2609.24144

LLM post-trainingGRPOReinforcement LearningStochastic EnvironmentCommon random numbersTool-use agentsPre-registered

arXiv Code

2026

Clarification Forensics: Does Halting After Clarification Reveal Misunderstanding?

So far, telling Qwen3.5-9B the file is protected drops its protected-file picks from 27 of 36 to 1, but that alone doesn't show it was confused.

Ongoing

AI safetyInterpretabilityModel forensicsQwen3.5-9B

Code

2025Peer-reviewed workshop paper

MemAgent: A Cache-Inspired Framework for Augmenting Conversational Web Agents With Task-Specific Information

MemAgent asks for task details once and caches them with an expiry date, so recurring web tasks took 58% less time in a 15-person study.

Supervised by Prof. Dr. Anindya Iqbal

Mentored by Faria Huq, Abhik Bhattacharjee

Published WOA 2025, CEUR Workshop Proceedings Vol. 4028

Web agentsAgent memoryConversational agents

Paper Code

Research experience

Role Research Intern

Program UIUC++ Summer Research in Software Engineering

Dates Summer 2026

Collaborated with Prof. Darko Marinov (UIUC),Prof. Tianyin Xu (UIUC) and Prof. Wenxi Wang (University of Virginia)

Work

Turning Failure Postmortems into SRE-Benchmark Problems
Ongoing work on two projects: a Verus project and a survey with accompanying experiments. (Ongoing)

Research interests

AI for Software Engineering and SRE

Benchmarks, verifiers and testing

LLM Post-Training

Critic-free RL and reward design

AI Safety

Interpretability and model forensics

Agent Orchestration

Multi-agent workflows and tool use

Natural Language Processing

Low-resource and multilingual settings

Human–Computer Interaction

Agent–human collaboration interfaces

Open to collaboration

I'm actively looking for research collaborations in AI for software engineering, LLM post-training and AI safety. If you're working on something adjacent, I'd like to hear about it.

Get in touch

Experience

Where I've built things

Six roles across AI research, applied LLM engineering and full-stack product work, from founding-engineer seats at early-stage ventures to teaching at university.

BRAC University Current

Dhaka, Bangladesh
Lecturer
Present
- Instructing undergraduate courses including Machine Learning, Operating Systems, Software Engineering, and System Analysis & Design.
- Mentoring students on architectural patterns and bridging the gap between theoretical concepts and industry application.
Sluqe

Warszawa, Poland (Remote)
AI/ML Engineer
Apr 2025 – May 2026
- Sluqe: built a voice-notes platform that transcribes, structures and makes recordings searchable and chattable. Delivered the Next.js dashboard and Flutter iOS/Android app on Firebase Cloud Functions and Firestore, with Genkit agent flows for chat over personal notes, web search, deep research, folders, share links, Notion export and Stripe billing.
- IoT SOS Alert System: built a device-agnostic emergency pipeline with a Terraform-provisioned serverless backend and an asyncio TCP bridge. Integrated Tuya WiFi buttons, binary 4G terminals and GPS bracelets through protocol adapters, escalating alerts to registered contacts through SNS and Twilio. Covered the system with unit tests, and integration tests.
- Aviators Spring: developed a Flutter app for pilots with Mapbox layers for airports, weather and VFR sectional charts. Rebuilt FAA NASR ingestion as a two-tier GeoJSON pipeline and automated the parallel publication of roughly 3.5 GB of FAA sectional GeoTIFFs as Mapbox tilesets.
Intellesphere

Dhaka, Bangladesh (Remote) BUET-affiliated venture
Founding Engineer (Backend & AI)
Mar 2024 – Mar 2025
- Pubali–Intellesphere: engineered a banking RAG assistant handling complex PDF and table parsing with LlamaParse and Qdrant hybrid indexing, serving both public bank data and personalised customer queries with a human-in-the-loop escalation path. Secured access with Keycloak, authentication with Supabase, and analytics with Metabase.
- Shajgoj-AI: built a multimodal product search engine over text and image inputs using CLIP embeddings and ChromaDB, with reranking and a recommendation layer. Collected the catalogue via Scrapy and Playwright, and ran the pipeline on Celery with Redis caching behind FastAPI.
- CivilMate: developed a civil-engineering automation tool using multimodal reasoning to compare design drawings against construction site imagery, automating project logging, blocker detection and stakeholder notification.
- Titas Gas: shipped a RAG chat platform blending an internal knowledge base with real-time web results via the Serper API.
- Optimised RAG pipelines end to end, including query normalisation, knowledge integration and reranking.
Priyo

Dhaka, Bangladesh
Software Engineer (Backend)
Jun 2024 – Dec 2024
- Priyo-Penguin: deployed an AI customer-support assistant on Django, Nginx and the OpenAI API, handling concurrent real-time queries against a curated knowledge base built with LlamaIndex, hybrid indexing and a vector store.
- Dolphin-Priyo: built a CRM and data-warehousing application unifying user management across Priyo platforms, with REST APIs in Django and PostgreSQL, Docker containerisation, an Nginx reverse proxy and a CI/CD pipeline.
- Shipped Priyo Play (play.priyo.com), a platform for quizzes, contests and games.
- Implemented bot protection with Cloudflare Turnstile, dynamic email campaigns via SendGrid and Twilio, and business dashboards in Metabase.
Oleyn

Dhaka, Bangladesh (Remote)
Founding Engineer (Backend & AI)
May 2024 – Dec 2024
- Shrobon: engineered an AI-powered CX and marketing-campaign system for large customer centres. Built a Bengali ASR stack with fine-tuned models and NVIDIA NeMo speaker diarization for low-latency transcript generation, fronted by a RAG assistant on FastAPI with Redis caching and Docker deployment.
- SuperAttorney: architected a legal research assistant for high-speed document parsing and retrieval, streamlining research, drafting and decision-making for legal professionals.
Yobo

Ontario, Canada (Remote)
Software Engineer Intern
Jan 2024 – May 2024
- Built YoboSQL, a text-to-SQL chat interface over large live SQL and NoSQL databases using LangChain, FastAPI and SvelteKit.
- Optimised database interactions against MariaDB and handled deployment on cPanel environments.

Projects

Selected work

Research pipelines, production AI systems and award-winning hackathon builds. Metrics are reported as measured, including the run that did not work.

17 projects

DC-Ops Assistant

Personal research project

+188%
reward gain over 7 epochs: 1×
AMD MI300X

A physics-grounded, three-phase training pipeline that fine-tunes Qwen2.5-7B-Instruct for datacenter NOC operations. Synthesised a 7,107-turn teacher-distilled SFT dataset with DeepSeek-R1-Distill-Qwen-32B against a custom RC-thermal simulation environment, then applied GRPO with a composite reward spanning format compliance, live physics feedback, scenario-aware heuristics and anti-looping penalties.

UnslothTRLvLLMGRPOOpenEnvTransformersHugging FaceLLM-as-a-judge +2

Code Live Space

Scout

Personal project · agentic systems

1.00
precision and recall, detector gate in CI: 197
tests against a real Postgres

An autonomous competitive intelligence system that monitors competitor pricing, scores meaningful changes, and produces source-linked weekly briefs. A three-tier escalation pipeline keeps costs low, while transactional job claiming and bitemporal facts ensure reliable processing and historical price tracking.

FastAPINext.jsLangfuseDockerBright DataPlaywrightTerraformPostgreSQL +6

Code Video demo

Medical CoT Assistant

Personal research project

20B
parameters fine-tuned: INT4
quantised for deployment

Fine-tunes a 20B-parameter model on curated medical reasoning datasets to produce a clinical assistant capable of chain-of-thought reasoning. The pipeline merges and reformats multiple medical QA corpora, applies LoRA fine-tuning with Unsloth, quantises weights to INT4 for efficient deployment, exports to GGUF, and evaluates with an LLM-as-a-judge harness.

Unslothllama.cppTransformersHugging FaceLoRAGGUFLLM-as-a-judgeROCm

Code Video demo

GeoQL-4B

Personal research project · negative result

68.49
OQS after SFT: +0.23
OQS after GRPO

Text-to-OverpassQL model fine-tuned from Qwen3-4B-Instruct-2507 with rank-64 LoRA on 6,352 OverpassNL examples, followed by GRPO with five reward signals. The RL stage improved OQS by only 0.23, within noise. The repository documents the negative result and its likely causes: reward redundancy, a knowledge bottleneck, decoding mismatch, weak proxies and collapsed learning signal.

Qwen3-4BLoRA r64TRLUnslothGRPOvLLMModalOverpassQL +1

Code Model card

CivilMate

Personal project · ongoing

An AI assistant for civil engineers built on graph RAG and agentic retrieval, answering questions about building codes and generating site reports from captured imagery. Actively iterating on report-generation quality and image analysis.

FastAPIOpenAI APINeo4jQdrantSupabaseAlembicPostgreSQLSQLAlchemy +2

Code Live site

তক্ষক (Tokkhok)

KUET BITFEST Hackathon 2025

A personalised RAG chatbot platform whose hard problem was Banglish, Bengali written in Latin script, which had to be transliterated to Bengali before retrieval. Supports bot sharing, behaviour customisation and few-shot inference.

FastAPIOpenAI APIGoogle Transliterate APIQdrantSupabaseAlembicPostgreSQLSQLAlchemy

Code Video demo

InsightAI

BCOLBD AI Hackathon 2024

An AI-powered productivity suite for students, combining retrieval over course material with real-time web search and task management to consolidate academic and personal workflows.

FastAPISpring BootSvelteKitOpenAI APILlamaIndexRAGSerper APIQdrant +5

Code Video demo

BD Prescription OCR

Personal research project

92.53%
test accuracy: 0.9234
macro-F1 (+6.44 pts over baseline)

A 78-way handwritten medicine-name classifier on the Doctor’s Handwritten Prescription BD dataset. Built a ResNet50-CRNN that runs a bidirectional LSTM over backbone feature maps to preserve left-to-right word structure, then ensembled a 5-fold StratifiedGroupKFold SWA run with hash-based leakage detection.

PyTorchResNet50CRNNSWAscikit-learnMLflowGrad-CAMCLAHE

Code Artefacts

ShopGenie

Lablab AI – Reasoning with o1, 2024

An AI shopping platform over 21,000 scraped products supporting multimodal search by text, image or both, with a recommendation system and sentiment analysis on reviews.

OpenAI APICLIP ViTSeleniumFastAPIQdrantSvelteKitTypeScriptPostgreSQL +1

Code

CoderHub

Code Crafters Dev Sprint 2024 – 1st place

A developer collaboration platform with project management (upload, diff, download), video calling, collaborative coding, groups, skill-based project matching and an AI chat assistant.

SvelteKitAgora SDKMonaco EditorTypeScriptPostgreSQLSupabaseTailwindCSSDaisyUI

Code Video demo

BDeHR

Therap JavaFest 2023

A healthcare records platform giving patients and doctors shared access to medical history, designed around data privacy and access control.

SvelteSpring BootJavaMySQLTypeScriptTailwindCSS

Code Video demo

Voyager

IUT OpenAPI Hackathon 2024 – 1st place

An attraction-discovery platform surfacing weather, amenities and reviews for locations, with user-contributed travel experiences and suggestions.

SvelteKitTypeScriptPostgreSQLDrizzleGeoApifyGeolocationTailwindCSSDaisyUI

Code

NerdHerd

CSE 408 – Software Development project

A distraction-minimising productivity app bundling collaborative note-taking, video calling, article writing, real-time quizzes and an AI chatbot into one workspace.

SvelteKitTypeScriptPostgreSQLSupabaseAgora SDKRealtime APITailwindCSSDaisyUI

Code

BebshaAI

SUST CSE Carnival Hackathon 2024 – 1st place

An AI platform helping small and midsize enterprises establish an online presence, generating a storefront where they can list and sell products with minimal setup.

ReactSpring BootFlaskPythonOpenAI APIOpenCVMaterial UI

Code

Vault

Personal project · resilient file transfer

A private file manager that lets approved users upload large folders directly into the owner’s Google Drive. Its split control and data planes bypass the server for file bytes, while resumable chunks, IndexedDB queue persistence, adaptive chunk sizing, checksum verification and deduplication keep terabyte-scale transfers reliable across dropped connections, browser crashes and server restarts.

FastAPIReactGoogle Drive APIIndexedDBSQLiteDockerCaddyGCP +1

Code

Scraping Toolkit

Dataset engineering

A collection of large-scale scraping pipelines covering extraction, cleaning and dataset construction, built to produce RAG-ready corpora with several datasets published to Hugging Face.

ScrapyPlaywrightSeleniumScrapeOpsBeautifulSoupHugging Face

Datasets

Subdomain Enumeration

CSE 406 – Computer Security lab

A subdomain discovery and assessment toolchain that enumerates the subdomains of a target domain and probes their authentication surface, built as a controlled security lab exercise.

Sublist3rScoutBurp SuiteHatch

Video demo

Blog

Research, in plain words

Short write-ups of my papers and projects: what we asked, what we found, and the figures that tell the story.

23 Oct 2026 10 min read

Audit the group, not the patch: full-suite rewards for coding agents at a fraction of the cost

Coding agents learn from test results, but training pipelines often run only the tests the original fix touched. In repositories with large test suites, about 1 in 22 patches that pass those tests break code elsewhere, and smarter test selection misses many of them. Randomly auditing whole groups of attempts recovers the full suite’s verdict without bias, at under a quarter of the test time.

Reinforcement learningGRPOCoding agentsSoftware testingRegression testingTest oraclesPre-registered

17 Oct 2026 10 min read

Green checks, broken clusters: testing the graders of AI SRE benchmarks

AI agents are now ranked on how well they fix broken Kubernetes applications, and a grading script decides every result. I tested the graders of four such benchmarks. They passed clusters that were still broken, failed fixes that worked, and the usual sanity checks missed most of it.

Agent evaluationBenchmarksSoftware testingTest oraclesKubernetesAI agents

12 Oct 2026 8 min read

Seen but not bound: AI agents treat a safety rule as a dial, not a gate

We gave LLM operations agents one rule: ask before doing anything destructive in production. They spotted production and judged the risk. Then they turned their caution up on every ticket instead of stopping where the rule applies.

Mechanistic interpretabilityModel internalsAgent evaluationAI agentsAI safetyBenchmarks

1 Oct 2026 10 min read

Skipped tests tell no tales: the hidden cost of test selection in CI

CI runs every test on every change, and test selection promises to skip most of them safely. But a selector in production never learns how the tests it skipped would have done, while the replays used to judge it quietly tell it. Replayed honestly, a leading history-based selector let nearly three times as many failing builds slip through, while selectors that read the change barely noticed.

Software testingContinuous integrationTest selectionEvaluation methodologyLLMsPre-registered

9 Sep 2026 11 min read

Luck is not skill: giving every rollout the same bad luck

Group-relative RL trains an LLM agent by comparing attempts at the same task, but a tool outage or a grading slip can make luck look like skill. I gave every attempt in a group the same luck. Comparisons got cleaner and gradient noise dropped at every checkpoint, and under tool faults the agent finished five points ahead, but the learning-curve tests I registered in advance did not pass.

Reinforcement learningGRPOLLM agentsStochastic environmentTool-use agentsCommon random numbersPre-registered

16 May 2026 10 min read

When the cooling fails: teaching a 7B model to run a datacenter

Datacenter operators keep cooling and power alive by reading a dashboard and typing the right command. I built a physics simulator of that job and trained a 7B model in it, first by imitating a larger reasoning model and then with reinforcement learning against the live simulation. It went from waiting to acting, but it never resolved a hard incident, and a closer look at the evaluation changes what its numbers can claim.

Reinforcement learningGRPOLLM agentsKnowledge distillationAgent evaluationPhysical AI

Research

Open source

News

Passing Without Repairing: Verifier Blind Spots in a Live SRE Agent Benchmark

Seen but Not Bound: LLM Operations Agents Treat Confirmation Policies as a Caution Dial, Not a Gate

FaithfulCut

Luck Is Not Skill: When Do Paired Rollouts Help Group-Relative RL of LLM Agents?

Clarification Forensics: Does Halting After Clarification Reveal Misunderstanding?

MemAgent: A Cache-Inspired Framework for Augmenting Conversational Web Agents With Task-Specific Information

Research experience

Research interests

Open to collaboration

Lecturer

AI/ML Engineer

Intellesphere

Founding Engineer (Backend & AI)

Software Engineer (Backend)

Founding Engineer (Backend & AI)

Software Engineer Intern

Bangladesh University of Engineering and Technology

Notre Dame College

Cumilla Zilla School

DC-Ops Assistant

Scout

Medical CoT Assistant

GeoQL-4B

CivilMate

তক্ষক (Tokkhok)

InsightAI

BD Prescription OCR

ShopGenie

CoderHub

BDeHR

Voyager

NerdHerd

BebshaAI

Vault

Scraping Toolkit

Subdomain Enumeration

2024

IUT 11th National ICT Fest – OpenAPI Hackathon

Code Crafters Dev Sprint 2024

SUST CSE Carnival 2024 – Hackathon

Gen-Dev AI Hackathon – Gender Knowledge Augmentation

BCOLBD 2024 – AI Category

Lablab AI – Reasoning with o1

2023

Cefalo ITverse 2023 – Project Showcase

2022

SEC Inter-University Junior Programming Contest 2022

AI & Machine Learning

Languages

Backend

Frontend & Mobile

Data & Vector Stores

Infrastructure & Tooling

Cloud Platforms

Certifications

Machine Learning Specialization

Deep Learning Specialization

PyTorch for Deep Learning

Generative AI with Large Language Models

AI Agents Course

Django Specialization

Algorithmic Toolbox

The Complete JavaScript Course

Problem Solving (Intermediate)

SQL (Intermediate)