Forbidden Tomes: AI, Security, and Responsible Engineering

Some spells are dangerous not because they fail, but because they succeed too easily.

Every workshop eventually acquires a locked cabinet. The books inside are not necessarily evil, and the spells written across their pages may be extraordinarily useful. They are locked away because their power changes the consequences of carelessness. Artificial intelligence has reached a similar place in software engineering. The same tools that can explain unfamiliar code, generate tests, draft documentation, analyze failures, and accelerate development can also expose confidential information, introduce vulnerabilities, recommend questionable dependencies, or quietly move protected intellectual property beyond boundaries an engineer never intended to cross.

Responsible AI engineering is therefore more complicated than deciding whether AI is good or bad. The useful question is whether we understand the boundaries surrounding the tool well enough to use it deliberately. A developer who refuses every AI assistant may avoid one category of risk while surrendering legitimate productivity gains. A developer who sends everything to an AI system without considering what leaves the organization may gain extraordinary speed while creating risks that remain invisible until much later. Experienced engineering lives between those extremes, where productivity is valuable but never exempt from judgment.

The lesson matters because many AI mistakes do not look dangerous while they are happening. Copying a production method into a prompt feels remarkably similar to copying it into a private scratch file. Asking an assistant to diagnose a database error feels like asking a colleague for help. Generating an authentication function may take only seconds, and accepting a suggested package may feel no different from following advice in documentation. The interface makes all of these actions easy, but ease is not evidence of safety. Convenience changes the cost of an action, not the responsibility attached to it.

The Restricted Wing

Security begins long before an attacker appears. It begins with understanding what information exists, who should have access to it, where it is allowed to travel, and what systems are permitted to process it. Software engineers already make these decisions when working with databases, logs, source repositories, production environments, monitoring platforms, and third-party services. AI does not eliminate those responsibilities. It introduces another destination that must be evaluated.

Consider what an engineer might casually include while asking an AI assistant to troubleshoot a production problem. A useful prompt could contain application code, a stack trace, a database query, customer information, configuration values, internal hostnames, API endpoints, business rules, or credentials accidentally captured in logs. Each piece provides context that may improve the answer, but each may also have different rules governing where it can travel. The engineer therefore has to evaluate the information before the model ever sees it.

The important distinction is not simply sensitive versus nonsensitive. Engineers should learn to think in terms of boundaries. Public documentation can usually cross far more boundaries than proprietary source code. Sanitized example data is different from customer records, just as a fictional API token is different from a production credential. An algorithm published in an open-source repository is different from a company’s unreleased business logic. Responsible AI usage begins by recognizing those distinctions before information enters a prompt.

This is familiar engineering territory. We already apply least privilege to users and services because access should be limited to what a task requires. A similar principle belongs in AI-assisted workflows: least context. Give an AI system enough information to perform the legitimate task, but do not provide information merely because copying the entire file, log, or dataset is easier. More context can produce a better answer, but unnecessary context can also create unnecessary exposure.

A Spellbook Is Still a System Boundary

One reason developers can underestimate AI risk is that conversational interfaces feel informal. A prompt box resembles a chat window rather than an integration point between systems. From an architectural perspective, however, sending information to an external AI service is still data movement. The friendly interface does not erase the boundary, and the engineer still needs to understand what is crossing it.

Imagine an engineer debugging an authentication failure. The fastest approach may appear to be pasting the relevant configuration directly into an assistant:

DATABASE_URL=postgres://production-user:password@internal-db/prod
JWT_SECRET=actual-production-secret
PAYMENT_API_KEY=actual-api-key

Our authentication service started returning 500 errors.
Here is the configuration and stack trace. What is wrong?

The engineering failure here has nothing to do with prompt quality. The prompt may be excellent for diagnosis. The problem is that the engineer has exposed credentials that should never have been included in the first place. Even if the AI system is approved for engineering work, unnecessary secrets have crossed a boundary they did not need to cross. The better question is not whether the tool can receive the information, but whether the task actually requires it.

A safer workflow changes the information before asking for assistance:

DATABASE_URL=[REDACTED]
JWT_SECRET=[REDACTED]
PAYMENT_API_KEY=[REDACTED]

Our authentication service started returning 500 errors
after a configuration change.

Relevant error:
Token validation failed during authentication initialization.

What categories of configuration problems should I investigate?

The second prompt may require another round of conversation, but that is often an acceptable trade. The engineer preserves enough context to obtain useful guidance while withholding secrets irrelevant to the reasoning task. The broader lesson is not merely to redact passwords. It is to develop the habit of asking what information the model genuinely needs and then supplying no more than that.

The Treasure in the Source Repository

Credentials are obvious treasures. Intellectual property is easier to overlook because developers work with source code all day. A production repository may contain proprietary algorithms, internal architecture, security controls, business processes, customer-specific logic, unreleased features, or material governed by contractual and licensing obligations. An engineer may have legitimate access to all of it without having permission to send it to every external service. Authorization to read information is not automatically authorization to redistribute it.

This distinction becomes particularly important when organizations use several AI tools with different deployment models and approved purposes. An enterprise-approved assistant operating within organizational controls may be suitable for material that should never be submitted to an unapproved service. An internally hosted model may establish another boundary entirely. Engineers cannot infer those permissions from the quality of the model, the convenience of the interface, or the fact that another team happens to use the same tool.

Responsible organizations therefore need clearer guidance than simply telling developers not to paste secrets into AI. Developers should know which tools are approved, which categories of information each tool may process, and what review requirements govern AI-assisted work. Security guidance consisting entirely of prohibition tends to fail because engineers still have legitimate problems to solve. Useful policy creates safe paths for accomplishing the work, much as a well-managed castle library distinguishes its public shelves from guild archives and restricted collections rather than chaining every book to the wall.

When the Spell Writes Back

Information flowing into an AI system is only half of the problem. Generated output also crosses a trust boundary when it returns. In the previous chapter of The Enchanted Workshop, we examined the danger of trusting convincing AI output merely because it looks authoritative. Security raises the stakes because generated code can compile, pass superficial tests, and still create a vulnerability. The most dangerous generated code may be code that works perfectly under ordinary conditions because successful execution encourages confidence.

Suppose an assistant generates a convenient endpoint for retrieving a customer record:

</> JavaSxript


app.get("/api/customers/:id", async (req, res) => {
    const customer = await db.customers.findById(req.params.id);

    if (!customer) {
        return res.status(404).json({ error: "Customer not found" });
    }

    res.json(customer);
});

The implementation is clean and plausible, and it may satisfy the request exactly as written. Yet an important security question is missing: should the current user be allowed to retrieve this particular customer record? A generated solution can correctly implement an incomplete requirement. That distinction matters because AI is often very good at solving the problem we describe, even when our description omits the security properties the production system actually needs.

The design must therefore make authorization part of the requirement rather than an afterthought:

</> JavaScript

app.get("/api/customers/:id", requireAuth, async (req, res) => {
    const customer = await db.customers.findById(req.params.id);

    if (!customer) {
        return res.status(404).json({ error: "Customer not found" });
    }

    if (!canViewCustomer(req.user, customer)) {
        return res.status(403).json({ error: "Forbidden" });
    }

    res.json(toPublicCustomer(customer));
});

Even this example should not be treated as a universal implementation pattern. The engineering lesson is the sequence of questions behind it. Authentication establishes identity, authorization determines whether that identity may perform the requested action, and output filtering determines which fields may leave the service. AI can help implement those controls, but engineers must first recognize that the controls belong in the design.

The Curse of Successful Code

AI changes the economics of software development in an important way. Generating code has become cheaper, but understanding the consequences of code has not. A developer can move from requirement to plausible implementation in minutes, sometimes seconds, while the security properties of that implementation still demand the same careful reasoning they required before generative AI entered the workshop. Speed compresses the implementation phase, not the responsibility that follows it.

When implementation was slower, there was natural friction between an idea and a working feature. Developers encountered documentation, inspected APIs, discussed design decisions, and wrestled with implementation details along the way. That friction was sometimes frustrating, but parts of it also functioned as an accidental learning mechanism. AI can remove much of that friction, which is valuable, but it can also remove moments when developers would otherwise notice assumptions about authentication, data access, validation, or trust. A solution that arrives fully formed can look finished before the engineering thought process is finished.

The answer is not to restore inefficiency for its own sake. It is to replace accidental friction with deliberate review. AI-generated authentication code should receive authentication-level scrutiny, database queries should receive data-access scrutiny, and file handling should be examined for validation, path traversal, size limits, and permissions. Generated network calls should raise questions about destinations, timeouts, credentials, certificates, and untrusted responses. The engineer should be able to explain why the solution is safe, what assumptions it makes, and how it behaves when those assumptions fail.

This changes the role of code review in AI-assisted development. Reviewers need to understand the assumptions behind generated code, especially where security boundaries are involved. A green test suite can demonstrate expected behavior without proving that the implementation is safe under hostile or unexpected conditions. The useful question is therefore not whether the AI wrote good code. It is whether the engineering team can defend every consequential decision in the code it chooses to ship.

Summoning Dependencies

AI assistants also influence software through recommendations. Ask for a solution to a specialized problem and a model may suggest a library, package, service, or API that appears to solve it immediately. Sometimes the recommendation is excellent. Sometimes the package is obsolete, inappropriate, nonexistent, or simply not something that should enter a production dependency graph without investigation. The speed of the recommendation can make the dependency feel like part of the answer rather than a separate engineering decision.

A dependency is not merely a shortcut to functionality. It is code your system now trusts. It can introduce transitive dependencies, maintenance obligations, licensing requirements, vulnerabilities, update schedules, and another external project whose decisions can affect your application. That was true long before AI assistants existed, but AI can make adding dependencies easier than understanding them.

Before adopting an AI-suggested dependency, engineers should apply the same evaluation they would use for any unfamiliar component. Verify that the project actually exists through authoritative sources. Examine its official documentation and repository, maintenance activity, ownership, release history, licensing, security posture, and whether its functionality justifies the new dependency. Confirm that the package being installed is the package you intended to install, particularly when names are similar or when an AI assistant supplies an exact installation command.

AI makes this discipline more important because recommendations can arrive without many of the contextual signals engineers encounter while researching manually. Documentation exposes surrounding information, repositories reveal activity and issue history, and package registries expose versions and ownership. A conversational answer can compress those signals into a confident recommendation, leaving the engineer with less visible evidence on which to base a trust decision. Let AI help identify possibilities, but verify important facts independently before adding another component to the kingdom.

The Privacy Ward

Security and privacy overlap, but they are not interchangeable. A system can protect information from unauthorized attackers while still processing personal information in ways users, contracts, policies, or organizational rules did not permit. Responsible AI engineering therefore requires more than keeping credentials out of prompts. It requires engineers to consider whether particular information should be processed by a particular AI system at all.

That distinction matters when developers use AI to analyze logs, summarize support tickets, classify user feedback, generate test data from production examples, or diagnose customer-specific problems. Removing passwords does not automatically make the remaining information appropriate to share. Names, email addresses, account identifiers, financial information, location information, internal correspondence, and combinations of otherwise ordinary details may all deserve protection. A log file can look purely technical while still containing a surprisingly detailed portrait of a real person.

Data minimization offers a durable engineering principle. If a task can be performed without personal information, remove it. If a production example can be replaced with synthetic data, replace it. If the AI needs the shape of a record rather than its actual contents, provide the schema or a sanitized representative example. The goal is not to starve the model of useful context, but to distinguish useful context from convenient context.

Consider an engineer investigating why an order-processing function fails for certain customers. Sending complete customer records may produce a quick diagnosis, but the model may only need data types, field relationships, and the unusual values triggering the failure. A sanitized example can preserve the technical characteristics of the problem without preserving the identity of the person behind it. Teams can reinforce this practice with synthetic debugging datasets, approved prompt patterns, redaction utilities, and other workflows that make data minimization easier rather than relying on developers to improvise it during an incident.

The Guild Needs Rules

Individual judgment matters, but responsible AI usage cannot depend entirely on every developer independently discovering the same boundaries. Once AI becomes part of normal engineering work, organizations need policies specific enough to guide decisions without becoming so restrictive that developers simply work around them. A rule that says to use AI responsibly provides little practical guidance. A blanket prohibition can be equally ineffective when approved AI systems clearly provide legitimate engineering value.

Useful policy begins with classification. Teams should understand what information may be used with approved AI systems, what requires additional protection, and what should never be submitted. Public source code, proprietary source code, credentials, customer data, internal documentation, unreleased product information, regulated information, and third-party intellectual property may require different treatment. Those categories should connect to actual tools and workflows so that an engineer facing a production problem can make a decision without interpreting vague policy language.

Organizations should also define which AI systems are approved for particular purposes. Different tools can have different deployment models, administrative controls, contractual arrangements, and access mechanisms. Engineers should not have to reverse-engineer those distinctions every time they open a prompt window. The guild should identify which libraries are open to ordinary apprentices and which archives require additional protection before anyone needs a forbidden tome in the middle of an emergency.

Policy must also address what comes back from the model. AI-assisted code should pass the same testing, review, security analysis, dependency evaluation, licensing checks, and deployment controls expected of other production code. Using an AI assistant does not transfer accountability to the assistant. If an engineer approves a change, merges it, deploys it, or recommends it to a team, that engineer participates in the decision regardless of who or what produced the first draft.

Guardrails Instead of Locked Doors

Strong responsible-AI programs should not rely primarily on prohibitions. They should create guardrails that allow engineers to gain legitimate productivity while keeping ordinary work inside acceptable boundaries. Security that makes legitimate work unnecessarily difficult encourages bypasses, while productivity systems that remove every constraint eventually create incidents. Mature engineering seeks the path where developers can move quickly because the surrounding system has been designed to help them move safely.

Those guardrails can exist throughout the development environment. Secret-scanning tools can catch credentials, data protection controls can identify restricted information, and approved AI systems can establish clearer organizational boundaries than ad hoc accounts. Dependency scanning, static analysis, code review, automated testing, and security testing can examine generated code using many of the same mechanisms applied to human-written contributions. None of these controls removes the need for judgment, but each reduces the number of moments when safety depends entirely on someone remembering a rule.

The same philosophy can shape the way teams interact with AI. Reusable workflows can encourage developers to provide abstractions, schemas, sanitized logs, minimal reproductions, and synthetic examples instead of entire production artifacts. Internal guidance can show engineers how to obtain useful assistance while exposing only the context required for the task. Responsible AI then becomes part of the engineering workflow rather than a warning attached to it.

A good guardrail preserves useful motion while constraining dangerous motion. That is why balancing AI productivity with security and privacy is not really a contest between innovation and caution. It is a design problem. The objective is not to decide whether the workshop receives powerful magic, but to build a workshop capable of using that magic without setting fire to the library.

The Engineer Behind the Spell

Guardrails matter because AI changes the mechanics of development, but it does not change the fundamental responsibility of an engineer. We have always worked with abstractions that allow us to accomplish more than we could build ourselves. Compilers generate machine instructions we rarely inspect, frameworks execute thousands of lines behind a few method calls, and cloud platforms provision infrastructure through concise configuration. Engineering has never required understanding every mechanism at its lowest level. It has required understanding enough to know where trust belongs.

AI belongs in that tradition of powerful abstraction, but with an important difference. Its outputs are probabilistic, contextual, and capable of appearing authoritative even when important assumptions are missing. An engineer can allow AI to accelerate repetitive work, explore alternatives, explain unfamiliar systems, draft implementations, generate tests, or expose possibilities worth investigating. The engineer remains responsible for deciding which information enters the system, which output deserves trust, which assumptions require verification, and which decisions should never be delegated.

Responsible AI use is therefore not measured by how little we use it. It is measured by how deliberately we use it. The objective is to preserve the leverage these tools provide without surrendering the judgment that makes the resulting work engineering rather than generation. The forbidden tome is not dangerous merely because it contains powerful spells. It becomes dangerous when the wizard stops asking what those spells can affect.

A Practical Ritual Before Casting

Responsible engineering principles become useful when they survive contact with an ordinary Tuesday afternoon. Security policies are easy to agree with during training and harder to remember when production is failing, a deadline is approaching, and an AI assistant appears capable of solving the problem in thirty seconds. That is precisely when a small decision framework becomes valuable. Before giving information to an AI system, I want to know what I am sending, why the model needs it, whether the tool is approved to receive it, and whether I can remove anything without damaging the task.

The same discipline applies when information returns. Before using generated code, I want to understand what security boundary it touches, what assumptions it makes, and what happens when those assumptions are violated. Before installing a suggested dependency, I want independent evidence that the component exists, is appropriate, and deserves a place in the system. Before trusting an answer involving privacy, intellectual property, licensing, authentication, authorization, cryptography, or production access, I want verification that does not depend entirely on the model that produced the recommendation.

These habits do not need to become ceremonial overhead for every trivial interaction. Asking an assistant to explain two public language features does not require the same scrutiny as asking it to analyze proprietary production code. Risk should determine the depth of review. A useful mental model is to evaluate three things: input, authority, and output. What information am I giving the system, what authority am I allowing it to influence or exercise, and what am I planning to do with what comes back?

When the Familiar Gains Keys

The stakes increase considerably when AI moves beyond conversation and begins acting through tools. An assistant that can suggest a database query presents one level of risk. An AI agent that can execute that query against production presents another. The underlying intelligence may be similar, but the authority attached to it has changed. Security architecture must change with that authority.

Least privilege therefore becomes even more important as AI systems gain access to repositories, terminals, cloud environments, ticketing systems, databases, deployment pipelines, and other engineering infrastructure. An agent should not receive broad production permissions simply because broad permissions make automation easier. Its credentials should be scoped to the task, its actions should be observable, consequential operations should have appropriate approval boundaries, and its access should be revocable. The same principles we apply to users and services still belong here.

Engineers should also distinguish between reversible and irreversible actions. Generating a patch in a temporary branch is different from merging it. Preparing an infrastructure plan is different from applying it. Drafting a database migration is different from executing it against production. AI-assisted workflows become safer when systems deliberately preserve these boundaries and require greater confidence as the consequences of an action increase.

None of this represents distrust of automation. Mature deployment systems already operate this way. Continuous integration can run tests automatically without permission to deploy everywhere, and a deployment service may release to staging while production requires additional authorization. AI agents should inherit these established engineering principles rather than receive exceptional privileges because their capabilities feel new. The more capable the familiar becomes, the more carefully the guild should decide which keys it carries.

Responsibility Cannot Be Automated Away

There is a tempting idea hidden inside almost every generation of developer tooling: if the tool becomes good enough, perhaps some category of engineering judgment will no longer be necessary. AI intensifies that temptation because it does not merely automate syntax. It can discuss architecture, recommend security controls, critique code, and explain its reasoning in language that sounds remarkably like an experienced colleague. That fluency can make delegation feel safer than it actually is.

Capability, however, should not be confused with accountability. An AI system does not own the production service six months after the original prompt disappears into history. It does not explain an architectural decision during a security review or decide how an organization should respond when confidential information crosses an unintended boundary. Those responsibilities remain with people and organizations, which means professional ownership cannot depend on who typed the original implementation.

AI may propose, draft, analyze, and challenge assumptions. When consequential software enters production, someone must still understand why the decision was acceptable and accept responsibility for making it. Teams should therefore reward sound judgment rather than raw generation speed and make it normal to reject impressive output when its assumptions cannot be defended. Discarding an AI-generated implementation because its security properties are unclear is not wasted productivity. It is engineering.

The Price of Powerful Magic

Throughout The Enchanted Workshop, AI has served as our familiar, scribe, testing assistant, restoration tool, and occasionally unreliable librarian. Each role has demonstrated the same underlying pattern. AI can dramatically reduce the cost of producing an artifact, whether that artifact is code, documentation, tests, analysis, or an architectural suggestion. What it does not automatically reduce is the cost of being wrong, and security makes that distinction impossible to ignore.

A weak comment can be corrected and an incomplete test can be strengthened. Exposed credentials may require immediate rotation, confidential information may be impossible to retrieve once disclosed, and vulnerable production code can create consequences far beyond the seconds required to generate it. Responsible engineering therefore considers not only the probability of failure but also its consequence. The higher the consequence, the less appropriate it becomes to rely on convenience, assumption, or unverified output.

This principle predates generative AI and will survive whatever tools eventually replace today’s models. Powerful engineering tools deserve neither fear nor worship. They deserve boundaries proportional to what they can affect. The safest AI-assisted organization may therefore not be the one that uses the least AI, but the one that understands its information, tools, permissions, and responsibilities well enough to use AI confidently within deliberate boundaries.

Closing the Forbidden Tome

For individual engineers, that discipline begins with ordinary habits. Minimize the information provided to AI systems, protect credentials and confidential data, understand organizational rules before sharing proprietary material, and treat generated code as something that must earn trust through review. Verify unfamiliar dependencies independently, apply privacy principles to debugging and analysis, limit the authority granted to AI agents, and preserve human approval around consequential actions. None of these practices requires abandoning the productivity AI offers. They are how we make that productivity sustainable.

That is the deeper lesson of Dangerous Magic, this week’s theme in The Enchanted Workshop. The danger is not that AI is secretly waiting to betray the engineer. The danger is that powerful capabilities can become ordinary so quickly that we stop noticing when an ordinary action crosses an extraordinary boundary. Responsible AI engineering is the discipline of continuing to notice, even when the tool makes everything feel effortless.

Yet another danger waits beyond the restricted shelves. If AI can write our code, explain our systems, diagnose our failures, generate our tests, and suggest our architecture, there is a quieter question we eventually have to confront. What happens to the engineer when the familiar performs so much of the craft that the wizard no longer practices the fundamentals? Security protects the workshop from what powerful tools can expose, but expertise protects the workshop from something just as consequential: forgetting how to work when the magic is unavailable.

On Friday, The Enchanted Workshop continues with The Wizard Who Forgot Magic: Keeping Your Engineering Skills Sharp. We will leave the forbidden library behind and examine skill atrophy, engineering fundamentals, critical thinking, and the responsibility to remain capable even when AI can perform more of the work for us. The greatest danger may not be that the spell fails. It may be that the spell works so well that the wizard forgets why it works.