Constraint Engineering: Building Reliable AI Workflows
A staff engineer at Parental shares a framework for integrating AI into software delivery. The discussion covers the 'charter' concept, deterministic factory architectures, and the critical role of human oversight in maintaining code quality and junior developer growth.
The Shift to Constraint Engineering
The integration of AI into software development is moving beyond simple prompt engineering toward a disciplined approach known as constraint engineering. Ian Johnson, a staff engineer at Parental, argues that the primary challenge is not the model's capability, but the lack of determinism. To address this, teams must build layered systems that constrain AI behavior, ensuring that non-deterministic outputs align with strict engineering standards.
The Charter and Harness Framework
A central concept in this framework is the 'charter,' defined as the agent-specific context and configuration that dictates acceptable behavior. Unlike general context, the charter is a special agreement that maps to agent primitives and directly influences coding decisions. This is distinct from the 'harness,' which is the software layer that executes the agent. By separating these concerns, teams can promote deterministic checks from simple rules to hooks and finally to custom harness workflows, increasing control as the scale of automation grows.
Deterministic Factories for Reliable Delivery
Johnson describes the implementation of a 'software factory,' a Python-based queue system that dispatches AI agents for specific tasks. Crucially, the factory itself contains no inference; it is a deterministic distributed system with dead letter queues and retry logic. This architecture allows for parallel work without the complexity of inferential orchestrators. Each job has strict input and output criteria, ensuring that a task only proceeds when preconditions are met, such as passing tests or meeting definition-of-done standards.
Human Oversight and Quality Assurance
Despite the automation, human ownership remains paramount. All changes are assigned to human engineers, who are responsible for understanding the code, responding to alerts, and making judgment calls. The team uses a 'prove step' where agents must demonstrate end-to-end functionality before staging, reducing the burden on manual QA. This approach allows QA to focus on exploratory testing rather than regression, which is vital for complex products like insurance platforms.
Implications for Team Dynamics
The adoption of AI tools has altered team dynamics, particularly regarding code review and junior development. Pair programming is now seen as a critical tool for knowledge transfer, helping juniors learn senior-level patterns in an AI-assisted environment. The team maintains a trunk-based delivery model with short-lived pull requests, ensuring that AI-generated code is integrated quickly and reliably. This strategy balances the speed of AI with the rigor required for production-grade software.
Key insights
-
The 'charter' is a specialized configuration layer that acts as a binding agreement between the team and the AI agent, distinct from general context. It ensures consistent behavior by defining standards and acceptable practices that the agent must follow in every turn.
Impact: Standardizing agent behavior through charters reduces variability in code output and improves the reliability of automated workflows across different tools and models.
-
AI agents should be embedded within deterministic distributed systems, such as job queues, rather than relying on inferential orchestrators. This 'factory' approach encapsulates non-determinism and allows for reliable parallel processing with standard retry and dead letter mechanisms.
Impact: Using deterministic infrastructure for AI tasks increases system stability and allows for scalable parallel work without the complexity and cost of managing inferential orchestration logic.
-
Human ownership of code changes remains critical, even when AI generates the code. Engineers must be assigned to tasks to ensure accountability, facilitate debugging, and make judgment calls that require human context, such as regulatory compliance.
Impact: Maintaining human ownership prevents skill atrophy and ensures that teams retain the ability to respond to incidents and make strategic decisions that AI cannot independently handle.
-
Fast feedback loops in CI/CD are essential for validating AI-generated code. Short-lived pull requests and strict testing gates ensure that AI output is verified quickly, reducing the risk of introducing bugs into the main branch.
Impact: Rigorous CI/CD practices mitigate the risks associated with AI's non-deterministic nature, allowing teams to ship AI-assisted code with the same confidence as human-written code.
-
Pair programming with AI is a vital strategy for training junior developers. It allows juniors to observe senior-level patterns and decision-making processes in real-time, helping them bridge the gap to seniority in an AI-assisted environment.
Impact: Proactive investment in junior development through AI-assisted pair programming ensures a sustainable pipeline of skilled engineers, countering the risk of a talent gap in the industry.
Action items
-
Define a formal 'charter' for your AI agents that includes coding standards, architectural patterns, and acceptable practices. Ensure this charter is consistently applied across all interactive and automated sessions.
Impact: A well-defined charter reduces the need for repetitive prompting and ensures that AI output aligns with team standards, improving code consistency and review efficiency.
-
Implement a deterministic queue-based system for dispatching AI agents, rather than using inferential orchestrators. Include dead letter queues and retry logic to handle failures gracefully.
Impact: This architecture increases the reliability of AI workflows and allows for scalable parallel processing, reducing the complexity of managing multiple AI agents.
-
Maintain strict CI/CD practices with short-lived pull requests and automated testing gates. Ensure that AI-generated code passes all tests and end-to-end checks before merging.
Impact: Fast feedback loops and rigorous testing reduce the risk of introducing bugs and ensure that AI-assisted code meets production quality standards.
-
Assign all AI-generated changes to human engineers for ownership and review. Require humans to understand the code and make final decisions on deployment and incident response.
Impact: Human ownership ensures accountability and maintains the team's ability to debug and respond to issues, preventing over-reliance on AI for critical decisions.
-
Integrate AI-assisted pair programming into the onboarding and training process for junior developers. Use this practice to teach senior-level patterns and decision-making skills.
Impact: This approach accelerates junior developer growth and ensures that the team retains the skills necessary to manage and improve AI-assisted workflows.
Quotes
“The reason why I call it a charter is because it is like an agreement with your coding agent.”
“There is no agent in the factory. There's no inferential or there's no inference in the factory. That's all deterministic, good old distributed systems, standard cues.”
“What matters with the change getting out is who the owner is, not necessarily the author.”