A practitioner’s guide to cracking senior GenAI/LLM engineering roles — from RAG pipelines to multi-agent orchestration
I’ve been in AI/ML for eight years. In the last two, almost every interview I’ve sat in — whether for senior data science, ML engineering, or AI product roles — has shifted toward Generative AI. The questions aren’t theoretical anymore. Interviewers want to know if you’ve actually built something: a RAG pipeline that didn’t hallucinate, a multi-agent system that didn’t deadlock, an LLM evaluation suite that caught regressions before production did.
This article compiles the 40 questions I’ve encountered most frequently — grouped by topic, with concise but precise answers. If you’re preparing for a senior GenAI role, bookmark this. If you’re already in one, use it to pressure-test your mental model.
Section 1: LLM Fundamentals
Q1. What is the difference between a base model and an instruction-tuned model?
A base model is trained purely on next-token prediction over large corpora. It can complete text but won’t follow instructions reliably. An instruction-tuned model (e.g., GPT-4, Claude) is further fine-tuned on curated instruction-response pairs — often using RLHF or RLAIF — to align outputs to user intent. In production, you almost always use instruction-tuned variants unless you’re doing a very specific fine-tuning task from scratch.
Q2. Explain the attention mechanism in transformers and why it matters for LLMs.
Attention allows each token to “attend” to all other tokens in the sequence and compute a weighted sum of their value vectors. The key innovation is that the weights (attention scores) are learned through Query-Key dot products. This enables long-range dependencies that RNNs couldn’t capture efficiently. For LLMs, self-attention is what allows the model to resolve pronoun references, track context across thousands of tokens, and perform multi-step reasoning.
Q3. What is the context window, and what are the practical challenges of a large one?
The context window is the maximum number of tokens the model can process in a single forward pass. Larger windows (128k+ in GPT-4o, Claude 3.7) improve in-context learning but come with quadratic attention complexity — O(n²) in memory and compute. Practically, models also exhibit a “lost in the middle” problem [1], where retrieval accuracy degrades for information positioned in the center of a long context.
Q4. What is temperature, and how does it affect generation?
Temperature scales the logits before the softmax. At temperature = 0, the model always picks the highest-probability token (greedy). At temperature = 1, probabilities are unchanged. Above 1, the distribution flattens and outputs become more random. For factual tasks, use low temperature (0.0–0.3). For creative tasks, 0.7–1.0 is appropriate.
Q5. What is the difference between top-k and top-p (nucleus) sampling?
Top-k restricts sampling to the k highest-probability tokens. Top-p samples from the smallest set of tokens whose cumulative probability exceeds p. Top-p is generally preferred because it dynamically adapts the candidate set to the entropy of the distribution — at low-entropy moments, it considers fewer tokens; at high-entropy moments, more. This produces more coherent and contextually appropriate outputs.
Section 2: Retrieval-Augmented Generation (RAG)
Q6. What problem does RAG solve, and what are its core components?
LLMs have a knowledge cutoff and can hallucinate on specific facts. RAG grounds generation in retrieved documents, combining the LLM’s language ability with real-time or domain-specific knowledge. Core components: (1) a document ingestion pipeline with chunking and embedding, (2) a vector store for similarity search, (3) a retriever, and (4) the LLM generator that synthesizes a response from retrieved context.
Q7. How do you choose a chunking strategy?
This depends on document type and query nature. Fixed-size chunking (e.g., 512 tokens with 50-token overlap) is simple but ignores semantic boundaries. Semantic chunking groups sentences by embedding similarity. Hierarchical chunking creates parent-child relationships — retrieving a small chunk but sending the parent for full context. For legal or structured documents, structure-aware chunking that respects section headers usually outperforms token-based approaches [2].
Q8. What is hybrid search, and when does it outperform pure vector search?
Hybrid search combines dense (vector) retrieval with sparse (BM25/TF-IDF) retrieval, then re-ranks using Reciprocal Rank Fusion or a learned reranker. Pure vector search excels at semantic similarity but struggles with keyword-exact queries (e.g., product codes, names, IDs). Hybrid search outperforms both individually when your query distribution is mixed — which is almost always in enterprise settings.
Q9. Explain the difference between a reranker and a bi-encoder.
A bi-encoder encodes the query and document independently into fixed vectors and computes similarity via dot product — fast but coarse. A reranker (cross-encoder) takes the concatenated query+document pair and scores it jointly using cross-attention — much slower but significantly more accurate. Best practice: use a bi-encoder for fast candidate retrieval from a large corpus, then apply a cross-encoder reranker to the top-k results.
Q10. How do you evaluate a RAG pipeline?
Using the RAGAS framework [3], you evaluate across four dimensions: (1) Faithfulness — are the claims in the answer grounded in the retrieved context? (2) Answer Relevance — does the answer actually address the question? (3) Context Precision — is the retrieved context relevant? (4) Context Recall — does the retrieved context contain the needed information? In production, I track faithfulness and context precision most closely since those catch hallucinations and retrieval drift.
Q11. What is the “lost in the middle” problem in RAG?
Research by Liu et al. [1] showed that LLMs are better at using information that appears at the beginning or end of the context window. Information in the middle of a long context is disproportionately ignored. This matters enormously for RAG when you stuff many chunks into the prompt. Mitigations: rerank chunks to put the most relevant ones first, use a “stuffing with boundary tokens” approach, or reduce the number of retrieved chunks.
Q12. What are the failure modes of a naive RAG pipeline in production?
(1) Chunk granularity mismatch — chunks too large dilute signal; too small lose context. (2) Embedding model-query domain mismatch. (3) Retrieval without reranking — top cosine similarity ≠ top relevant chunks. (4) No guardrails against off-topic queries — the LLM will hallucinate when context is irrelevant. (5) No citation tracking — impossible to audit answers. Each of these requires explicit mitigation in a production-grade system.
Section 3: Prompt Engineering
Q13. What is chain-of-thought prompting, and when does it help?
Chain-of-thought (CoT) prompting asks the model to reason step-by-step before producing the final answer [4]. It reliably improves performance on arithmetic, multi-step reasoning, and tasks requiring intermediate logic. It’s most effective with larger models (7B+ parameters) and when the reasoning chain is verifiable. For simple classification or retrieval tasks, CoT adds latency without benefit.
Q14. What is the difference between zero-shot, few-shot, and fine-tuning?
Zero-shot: provide only the task description. Few-shot: provide 2–8 demonstration examples in the prompt. Fine-tuning: adapt model weights on a task-specific dataset. In my experience, few-shot prompting closes ~70–80% of the gap between zero-shot and fine-tuning for most classification and extraction tasks. Fine-tuning is worth the operational overhead when you need consistent formatting, specialized vocabulary, or sub-100ms latency at scale.
Q15. What is prompt injection, and how do you defend against it?
Prompt injection occurs when user-provided input includes instructions that override the system prompt (e.g., “Ignore all previous instructions…”). Defenses: (1) delimit user input clearly from system instructions using XML tags or special tokens, (2) validate and sanitize inputs, (3) use a separate LLM call as an input classifier before the main chain, (4) apply output validation to catch unexpected behavior. Prompt injection is the #1 security vulnerability in LLM applications [5].
Section 4: Multi-Agent Systems
Q16. What is a multi-agent LLM system, and when should you use one?
A multi-agent system orchestrates multiple LLM-backed agents, each with specialized roles (researcher, coder, reviewer, etc.), that collaborate to complete complex tasks. Use multi-agent architectures when: (1) the task is too long to fit in a single context window, (2) the task benefits from parallel execution, (3) you need specialization and verification (e.g., one agent generates code, another tests it). Frameworks like CrewAI, LangGraph, and AutoGen provide orchestration primitives.
Q17. Explain the difference between ReAct and Plan-and-Execute agent architectures.
ReAct (Reason + Act) [6] interleaves reasoning traces and tool-use actions in a single loop — the agent thinks, acts, observes, repeats. Simple and effective for single-step tool use. Plan-and-Execute [7] separates planning from execution: first the planner generates a full task plan, then sub-agents execute each step. Plan-and-Execute handles long-horizon tasks better because the overall goal doesn’t get lost in iterative tool calls.
Q18. How do you prevent an agent from getting stuck in an infinite loop?
(1) Hard-limit the number of iterations/tool calls. (2) Implement a loop-detection heuristic (e.g., identical state or action sequence repeated N times). (3) Use a supervisor agent to monitor child agent progress and intervene. (4) Design termination criteria into the task decomposition prompt explicitly. (5) Set budget constraints on token usage or API calls.
Q19. How do agents share state in LangGraph?
LangGraph represents agent workflows as directed graphs where state is passed through a shared TypedDict state object. Each node (agent or tool) reads from and writes to this state. Conditional edges route execution based on state values. For multi-agent systems, you can isolate sub-graphs with their own state schemas and use message passing at the graph boundary — this prevents state pollution between specialized agents.
Q20. What is the role of memory in agent systems?
Agents need multiple memory types: (1) In-context memory — the current conversation/task history in the context window. (2) External episodic memory — a vector store of past interactions, retrieved as needed. (3) Semantic memory — a knowledge base of facts (often a RAG system). (4) Procedural memory — learned workflows or tool-use patterns, sometimes stored as fine-tuned weights. In production, managing what goes into the context window vs. external memory is a critical performance lever.
Section 5: Fine-Tuning and Alignment
Q21. When should you fine-tune vs. use RAG?
Use RAG when knowledge needs to be updatable, auditable, or domain-specific without retraining. Fine-tune when: (1) you need consistent output format or style, (2) the task requires skills not in the base model, (3) latency is critical and you can’t afford a retrieval step, or (4) you have 1,000+ high-quality labeled examples. Often the best architecture combines both: fine-tune for format/style, RAG for factual grounding.
Q22. What is LoRA and why is it preferred over full fine-tuning?
Low-Rank Adaptation (LoRA) [8] freezes the pretrained model weights and injects trainable rank-decomposition matrices into each transformer layer. This reduces trainable parameters by ~10,000x compared to full fine-tuning, enabling fine-tuning of 7B+ models on a single GPU. QLoRA [9] further quantizes the frozen weights to 4-bit, making it feasible to fine-tune 65B models on a single 48GB GPU. For most production use cases, LoRA/QLoRA provides 90%+ of the performance of full fine-tuning at a fraction of the cost.
Q23. What is RLHF, and what are its known limitations?
Reinforcement Learning from Human Feedback [10] trains a reward model on human preference data and then optimizes the LLM against it using PPO. It has produced the most aligned models we have. Known limitations: (1) reward hacking — the LLM learns to game the reward model rather than genuinely improve, (2) human labeler variance and bias, (3) expensive and slow to iterate, (4) distributional collapse if KL penalty is insufficient. DPO (Direct Preference Optimization) [11] has emerged as a simpler alternative that avoids explicit RL.
Section 6: LLM Evaluation and Observability
Q24. How do you measure LLM output quality in production?
A combination of: (1) automated LLM-as-judge scoring (using a stronger model to grade outputs), (2) task-specific metrics (ROUGE, BLEU for summarization; F1 for extraction; pass@k for code), (3) RAGAS metrics for RAG pipelines, (4) human evaluation for high-stakes use cases. Track metric distributions over time — sudden changes in score distribution often precede user complaints.
Q25. What is LLM-as-Judge, and what are its failure modes?
Using a strong LLM (e.g., GPT-4, Claude 3 Opus) to evaluate the outputs of a weaker or target LLM. It correlates well with human judgment at scale. Failure modes: (1) Position bias — judge tends to prefer the first option in a pairwise comparison, (2) Self-enhancement bias — a model judges its own outputs more favorably, (3) Length bias — longer outputs are often scored higher regardless of quality. Mitigations: randomize option order, use ensemble judging, control output length.
Q26. What observability tools do you use for LLM systems in production?
LangSmith (for LangChain-based systems), Phoenix (Arize), Langfuse, and PromptLayer are common choices. Key things to trace: prompt and completion content, token counts, latency, tool call results, retrieval scores, and final output quality scores. Set up automatic alerts for latency spikes, high token consumption, and LLM-as-judge score drops.
Section 7: Architecture and System Design
Q27. Design a production RAG system for a 10,000-document enterprise knowledge base.
Key design choices: (1) Ingestion — parse PDFs/Docx with structure-aware chunking (LlamaParse or Unstructured.io), embed with text-embedding-3-large or a fine-tuned E5 model. (2) Storage — Pinecone, Weaviate, or pgvector depending on scale. (3) Retrieval — hybrid search (BM25 + dense) with a cross-encoder reranker. (4) Generation — structured output with citations, a faithfulness check via RAGAS, and a fallback to “I don’t know” when context score is below threshold. (5) Observability — trace every query in LangSmith; alert on faithfulness score drops.
Q28. How would you handle a multi-tenant RAG system where each tenant’s data must be isolated?
Use namespace or metadata filtering at the vector store level — Pinecone namespaces or Weaviate tenant isolation are purpose-built for this. Never allow cross-tenant retrievals by filtering on a tenant_id metadata field at query time. For very sensitive data, use separate vector store indices per tenant (more expensive but zero cross-contamination risk). Combine with role-based access control on the API layer.
Q29. How do you reduce LLM latency in a production chatbot?
(1) Streaming — stream tokens to the client immediately. (2) Model right-sizing — use a smaller, faster model for simpler queries and route complex ones to a larger model. (3) Prompt caching — cache the system prompt prefix (Anthropic and OpenAI both support this). (4) Async tool calls — run independent tool calls in parallel. (5) Quantized inference — use 4-bit or 8-bit quantized models for self-hosted deployments.
Q30. What is guardrailing in LLM applications?
Guardrails are validation layers that sit before and after the LLM call to ensure inputs and outputs meet safety and policy requirements. Input guardrails: topic classifiers, PII detectors, injection detectors. Output guardrails: hallucination detectors, format validators, toxicity filters. NeMo Guardrails and Guardrails AI are popular frameworks. In regulated industries (healthcare, finance), output guardrails are non-negotiable.
Section 8: NL-to-SQL and Structured Data
Q31. What are the main failure modes of NL-to-SQL systems?
(1) Schema ambiguity — same concept named differently across tables. (2) Multi-hop joins the LLM reasons incorrectly about. (3) Hallucinated column/table names. (4) Incorrect aggregations (SUM vs. COUNT). (5) Date/time arithmetic errors. Mitigations: provide schema context with examples, use few-shot demonstrations from the target schema, add a SQL validation layer, and implement execution-based feedback where the generated SQL is executed and errors are fed back to the LLM for self-correction.
Q32. How do you handle large schemas in NL-to-SQL?
A large schema won’t fit in the context window. Use a two-stage retrieval: first retrieve the most relevant tables using the user query (embedding-based), then pass only those table DDLs and sample rows to the LLM for SQL generation. Tools like Vanna.ai implement this pattern. For very large schemas, train a schema understanding module on synthetic query-schema pairs.
Section 9: Knowledge Graphs and Structured Retrieval
Q33. When would you use a knowledge graph instead of a vector store for retrieval?
Knowledge graphs excel when: (1) relationships between entities are semantically important (e.g., “who reports to whom”), (2) multi-hop reasoning across entities is needed, (3) you need precise, auditable provenance. GraphRAG [12] by Microsoft combines graph traversal with vector retrieval and significantly outperforms naive RAG on community-level questions that require synthesizing information from multiple entities.
Q34. What is GraphRAG, and how does it differ from standard RAG?
GraphRAG builds an entity-relationship graph from the document corpus during indexing. At query time, it retrieves relevant graph communities (clusters of related entities) rather than individual chunks. It then summarizes these communities and uses them as context. The result: dramatically better performance on global, synthesis-type questions (e.g., “What are the main themes in this corpus?”) at the cost of higher indexing time and complexity [12].
Section 10: Rapid-Fire Conceptual Questions
Q35. What is the difference between semantic search and keyword search? Semantic search uses dense embeddings to find conceptually similar content. Keyword search (BM25) matches on exact or stemmed term overlap. Semantic search handles synonyms and paraphrasing better; keyword search handles proper nouns and exact IDs better.
Q36. What is function calling in LLMs? A mechanism where the LLM returns a structured JSON object specifying which function to call and with what arguments, instead of (or in addition to) natural language. Enables reliable tool use and is the foundation of most agent frameworks.
Q37. What is Constitutional AI (CAI)? Anthropic’s approach to alignment [13] where the model is trained to critique and revise its own outputs against a set of principles (“constitution”), reducing reliance on human labelers for harmful content.
Q38. What is speculative decoding? A latency optimization where a small draft model generates multiple tokens in parallel, and the larger target model verifies them in a single forward pass. Effective speedup is 2–3x for long generations.
Q39. What are the key differences between LangChain, LlamaIndex, and LangGraph? LangChain: general-purpose LLM application framework with broad integrations. LlamaIndex: specialized for data ingestion, indexing, and RAG. LangGraph: a low-level graph execution engine for stateful, multi-step agent workflows — best choice when you need full control over agent state and flow.
Q40. What is the difference between online and offline evaluation of LLMs? Offline: evaluation on a fixed benchmark dataset before deployment (fast, reproducible, but may not reflect production distribution). Online: evaluation of production traffic via sampling and LLM-as-judge scoring (reflects real user behavior but slower to iterate on). Best practice: maintain a curated regression suite for offline + monitor online metrics post-deployment.
Final Thoughts
These 40 questions reflect what’s actually being tested in senior GenAI and LLM engineering interviews right now. The field is moving fast — architectural patterns that were cutting-edge in 2023 (naive RAG) are considered baseline today, and interviewers are already probing into agentic memory, GraphRAG, and multi-modal pipelines.
The single biggest differentiator I’ve seen in interviews isn’t knowing the definitions — it’s being able to reason about trade-offs. Why hybrid over vector-only? When does fine-tuning beat prompting? Why LangGraph over LangChain? If you can answer those with real production experience behind them, you’ll stand out.
Want to Go Deeper?
I’ve compiled a comprehensive GenAI Interview Prep Guide — 80+ questions with in-depth answers, architecture diagrams, and a 2-week study plan — as a downloadable resource.
👉 Get the Full GenAI Interview Prep Pack on Gumroad
References
[1] Liu, N. F., et al. (2023). Lost in the Middle: How Language Models Use Long Contexts. arXiv:2307.03172.
[2] Langchain Blog. (2023). Evaluating RAG Architectures on Benchmark Tasks. langchain.com.
[3] Es, S., et al. (2023). RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217.
[4] Wei, J., et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS 2022.
[5] OWASP. (2023). OWASP Top 10 for Large Language Model Applications. owasp.org.
[6] Yao, S., et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629.
[7] Wang, L., et al. (2023). Plan-and-Solve Prompting. arXiv:2305.04091.
[8] Hu, E., et al. (2021). LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685.
[9] Dettmers, T., et al. (2023). QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023.
[10] Ouyang, L., et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.
[11] Rafailov, R., et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023.
[12] Edge, D., et al. (2024). From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Microsoft Research. arXiv:2404.16130.
[13] Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. Anthropic. arXiv:2212.08073.
If this was useful, follow me here on Medium for more practitioner-level GenAI content. I write about RAG systems, multi-agent architectures, and lessons from building AI products in production.
40 Generative AI Interview Questions That Actually Get Asked in 2026 (With Answers) was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.