RedCare executive warns AI coding creates unmanageable cognitive debt
Ilya Sakharov argues that AI-assisted development is eroding human ownership of software systems. He contends that the volume of non-deterministic code generated by large language models exceeds the rate at which engineers can comprehend or secure it.
Ilya Sakharov, Executive Director of Engineering at RedCare Pharmacy, asserts that the industry is accumulating "cognitive debt" rather than merely technical debt. Sakharov explains that by outsourcing the central reasoning process of development to large language models, engineers are creating systems they do not fully understand. He distinguishes this from traditional automation, which was designed to produce deterministic results from error-prone human inputs, whereas large language models are inherently non-deterministic. This shift creates a "black box" that is difficult to own because ownership requires the ability to influence and control the underlying architecture.
Sakharov warns that the verbosity and complexity of AI-generated code will soon make it impossible for humans to comprehend the systems they deploy. He predicts a future of "disposable code" where software is continuously regenerated rather than maintained through traditional continuous integration and deployment pipelines. This trend could lead to machine-to-machine languages that exclude humans from the development process entirely. The security implications are equally severe. Sakharov notes that while the rate of vulnerability discovery has increased by a factor of 100, the production of buggy and vulnerable software by large language models is growing exponentially faster. This imbalance degrades the overall security posture of the software landscape.
To address these challenges, Sakharov advocates for a shift toward "internal offensive" security integrated into the software development lifecycle. He argues that reactive measures like bug bounties are insufficient. At RedCare, engineering acts as the gatekeeper for AI adoption across non-technical departments, using an internal gateway to manage models from OpenAI and Anthropic. Sakharov emphasizes the need for "forward deployed engineers" to help other teams use AI within strict security and privacy guardrails. He also highlights a gap in financial measurement, arguing that current metrics like token consumption fail to capture the actual business value of AI usage.