AI should be owned, Not rented
AI Should Change Productivity, Not PeopleContinue reading on Medium »
Artificial Intelligence, llm-agent, llm-applications, llm-evaluation, retrieval-augmented-gen
llm-evaluation
Arize Agent Skills, claude-skills, Evals, evaluation framework, Evaluations, llm-evaluation, LLMs, model-evaluation, prompts, system prompt assembly
Models got an order of magnitude better at following instructions in one year
A year ago, frontier models started losing track of instructions somewhere around 200–300 simultaneous constraints. With 2026 models, that ceiling is closer to 2,000 — an order-of-magnitude jump. We re-ran IFScale to see how, and how each model fails.
The post Models got an order of magnitude better at following instructions in one year appeared first on Arize AI.
agent-harness, ai-system-architecture, evaluation framework, evaluation harness, evaluation-driven development, Evaluations, harness-engineering, llm-evaluation
What is an evaluation harness?
An evaluation harness is the standardized infrastructure that decides what gets evaluated, runs the evaluation, and acts on the result.
The post What is an evaluation harness? appeared first on Arize AI.
ai, data-security, enterprise-ai, llm-evaluation, privacy
I Built a Failure-Mode Test For OpenAI Privacy Filter. The Model Card Was Right.
A small enterprise-style test showing why PII redaction needs domain rules, span cleanup, and in-domain evaluation before it becomes a production control.Most sensitive values get caught. The gap is the production problem. Image: AI-generated by the au…
evaluation framework, evaluation-driven development (EDD), Evaluations, llm-as-a-judge, llm-evaluation, prompt evaluation, regression testing for llms
Prompt templates as configs, not code
This post was written in April 2026. Cloud products, feature maturity, and recommended patterns change over time, so readers should treat these examples as directional guidance. For teams already using Arize, there is a natural extension of that pattern. Prompt Playground can sit upstream of the config layer as the place where prompts are edited, compared, and versioned before they are promoted into whatever config system the company already trusts in production.
The post Prompt templates as configs, not code appeared first on Arize AI.
Artificial Intelligence, data-science, llm-evaluation, product-management, writing-prompts
The Eval Trap: Your Benchmark Is Part of Your Product
AI evals are becoming increasingly necessary and common, but improper benchmark design will fail to reveal how the system will behave in production, while giving you a false sense of stability. Below is the specific failure I encountered in this proces…
aiops, enterprise-ai, llm, llm-evaluation, mlops
Online Evals Done Right: Runtime Scoring and Review Queues for Production LLM Systems
A practical guide to online evals that score live traffic, apply LLM-as-judge checks, route risky cases to review, and feed production failures back into offline tests.Article 3 in a series on eval loops for production LLM systems, with a companion ref…
ai-ethics, Artificial Intelligence, llm-evaluation, red-teaming, women-in-tech
Is AI Grading Fair Testing Gender Bias in Essay Evaluation
Introduction: When Algorithms Become Microscopes for Social Bias
In an era of rapid artificial intelligence advancement, we often look to…Continue reading on Medium »
aiops, enterprise-ai, llm, llm-evaluation, mlops
How to Design Offline Eval Gates That Actually Catch Regressions Before Release
A practical guide to implementing offline release gates, with a reference implementation. Article 2 in a series on eval loops for production LLM systems.A release gate is not a benchmark report. It is a decision system.Most teams I’ve seen treat it lik…
Artificial Intelligence, llm-applications, llm-evaluation, Machine Learning
Our Benchmark Crashed Mid-Run. We Still Beat Industry SOTA.
What a live Dimensional Collapse taught us about the fragility of Vector RAG — and the resilience of Deterministic Spines.Continue reading on Medium »