hristosbilis.ai Search

Search

Try Merck & Co.validationAnnex 22Discovery

    hristosbilis.ai

    Framework

    AI Project Assessment Framework

    Decide how much checking a candidate AI use case needs, and whether it is worth building.

    Stage 2 of four: Discovery to Assessment to Solutioning to Adoption and Improvement. Discovery hands over a candidate and its baseline. A Go moves next to Solutioning.

    Open the AI project scorecard

    An AI idea has to earn a number before it earns a design. This page is the reasoning I use to decide which ideas get one. I work with Quality and AI leaders. You should be able to rebuild the framework from three ideas.

    1. Three ideas

    1. Value comes first. An idea with no measured result has nothing to set cost against. Every assessment starts with a value case. It needs a baseline measured before anything is built, a target, an owner who runs the check, and a check date. Set value against all lifecycle costs: implementation; assurance activities, including validation where applicable; licenses and operation; reviewer time; upkeep and change work; and retirement. No baseline means measure one before building.

    1. New output adds checking. That checking is a cost. A process change adds no new software or model output. It still needs an effectiveness check. Fixed rules (deterministic) give the same output for the same controlled input. Repeatability does not prove correctness. They still need risk-based testing against requirements and expected results. Generative models can give variable outputs. Define proportionate evaluation before deployment. For example, use an evaluation set: representative test cases with expected results. Monitor after deployment. Base both on intended use and risk. These activities can add cost. Prefer the tool that adds the least new checking work and still does the job.

    2. The work decides the bucket. When a result must be exact, prefer fixed rules or lookups where they can express the requirement. Generative AI may earn its added evaluation cost when the task depends on interpreting or drafting unstructured language and the outputs can be checked. Predictive, vision and other machine-learning uses need a separate assessment path. These four buckets do not classify them. Requirements and the quality system give formal responsibility to people and organizations, not software. For GxP (good practice) decisions with real consequences, an appropriately qualified, authorized person decides and approves before an output is sent.

    2. Four buckets, as a ladder

    Order the responses by the new checking work they add. Start with the smallest.

    1. Process fix. Change how the work runs. This adds no new software or model output. The process change still needs an effectiveness check.

    2. Light automation. Configured workflows for reminders, assignment and timing. Fixed, configured behavior can make expected results easier to specify. Assurance and audit-trail controls depend on intended use and documented risk assessment.

    3. Heavier automation without generative AI. Expert-authored logic, code and lookups, including control-chart and limit-based checks. Fixed rules can make expected results easier to specify. They still need risk-based testing against requirements.

    4. Generative AI. Bounded unstructured-language work that cannot be expressed adequately as a fixed rule.

    Key rule

    Stop at the lowest bucket that meets the targets. One deployment can combine buckets, with the model confined to a bounded language step. The bucket does not determine regulatory applicability or set validation scope.

    3. The questions, in order

    Each question asks whether a lower bucket is enough before you move up.

    1. Is the cause a broken process? Fix it first. Automating a broken process can keep or worsen its defects.
    2. Can the required behavior be specified with fixed rules? If yes, use Light automation for simple, stable configuration. Use Heavier automation without generative AI when expert-authored logic, code, or more substantial testing is needed.
    3. Does the remaining task require interpreting or drafting unstructured language, and is there a defensible way to test the outputs? If it is not language work, choose a lower bucket. If test data or acceptance criteria are missing, do not deploy until they exist.
    4. Does the design keep decisions that matter with appropriately qualified, authorized people, and control the harm of an error? For GxP decisions with real consequences, an appropriately qualified, authorized person decides and approves before an output is sent. If no, redesign and re-run Questions 2 to 5. If no acceptable design remains, stop.
    5. Does expected value exceed every lifecycle cost? Net the targets against the lifecycle cost of the design you landed on. A lower bucket that meets the same targets wins. A negative net, or one below the organization's defined threshold, is "Not this one."

    "Not this one" does not make the quality obligation go away. The task still needs a properly resourced solution.

    One rule outlasts the Go. It covers boundary creep. Evidence and controls for one intended use do not automatically cover another. When the use changes, document the impact assessment. Update controls before relying on the system.

    Figure

    Decision flow

    One vertical flow, exits to the right. Start: Value case. Baseline, target, owner, check date; set against all lifecycle costs. Question 1: Is the cause a broken process? Yes leads to Bucket 1, Process fix. No continues. Question 2: Can the required behavior be specified with fixed rules? Yes, simple and stable configuration, leads to Bucket 2, Light automation. Yes, expert-authored logic or code, leads to Bucket 3, Heavier automation without generative AI. No, or only part, continues. Question 3: Does the rest need unstructured-language work, and can outputs be tested? Yes leads to Bucket 4, Generative AI, bounded language step. Not language work: choose a lower bucket (Bucket 3 or earlier). Language work but no test data or acceptance criteria: do not deploy until they exist, then continue to Bucket 4. Question 4, reached from every bucket: Does the design keep decisions that matter with qualified, authorized people and control the harm of an error? No: redesign and re-run Questions 2 to 5; if no acceptable design remains, stop. Yes continues. Question 5: Does expected value exceed every lifecycle cost? Yes: Go, the lowest-cost option satisfying intended use, quality and compliance. No: Not this one, no-go for this design; fund the underlying quality work.

    4. Example: supplier change notices

    Made up. The company, use case, and numbers are for shape only. Dollar amounts marked "about" are rounded to the nearest $1,000.

    Suppliers email change notices to a regulated manufacturer. A coordinator reads each one, judges whether it touches the company's products, logs it, assigns an owner, and sends questions back. Someone proposes an AI assistant on the inbox.

    Speed is the median days to log and assign a notice. Compliance risk is the share logged after the 10-day internal target.

    Last quarter's baseline was 60 notices a month, median 6 days, 15 percent logged late, and 35 coordinator hours a month. The target is a median of 2 days, under 3 percent late, and 13 hours a month released. The owner is the supplier quality manager. The check date is day 90 after launch.

    Assume 13 hours released each month. Over three years, that is 13 × 12 × 3 = 468 hours. At an example loaded rate of $90 an hour, that is $42,120, shown as about $42,000. This is capacity value, not cash savings. The late-log reduction is forecast, not realized value.

    Walk the ladder.

    • Process fix: one mailbox, one form with required fields, one named owner.
    • Light automation: routing, acknowledgment, reminders at day 5 and day 8, escalation at day 10.
    • Heavier automation without generative AI: a lookup against the approved-supplier list and change types that always need quality review.
    • Generative AI: about 15 free-text notices a month remain. A narrow step qualifies. It drafts a summary and a supplier question for the coordinator to edit and send. The impact decision stays with an appropriately qualified, authorized person.

    This example uses a three-year horizon. Each three-year lifecycle cost is assumed to include build and integration and applicable assurance or validation. It also includes licenses and operation, human review, monitoring, maintenance and change assessment, and an allocated retirement cost.

    Option (example figures) One-time cost Yearly run cost Three-year lifecycle cost (assumed) Result against target
    A. Fix, form, routing, reminders and lookup (Buckets 1 to 3)Covers: Form notices $24,000 $4,000 $36,000 Predicted results for form notices only; about $42,000 capacity value
    B. Narrow drafting step added to A (incremental figures)Covers: 15 free-text notices a month $55,000 $22,000 $121,000 Assume 6 additional hours a month and a free-text baseline median of 3 days falling to 2; about $19,000 incremental capacity value
    C. AI assistant reads every notice and decides impactCovers: All 60 notices $120,000 $45,000 $255,000 Puts the impact decision on the model

    The decision: A is the right fix for notices that come in on the form, but it is not a Go yet.

    The targets cover all 60 notices a month. The table only predicts results for the form notices. So before approving A, estimate how all notices would do, and check that A's margin clears the organization's threshold. That margin is $6,120 of capacity value over A's $36,000 cost. At day 90, measure both targets again across all notices.

    B adds a generative AI drafting step for the 15 free-text notices. It qualifies as generative AI, but it is "Not this one" for now. It frees 216 hours over three years, worth $19,440, shown as about $19,000, against $121,000 of added lifecycle cost.

    C fails Question 4, because the model would decide impact. It goes back for redesign.

    What would change the decision. Run B's final check again when free-text volume or handling time makes B's added capacity value clear the organization's defined threshold against B's added lifecycle cost. Also run that check if a late log changes the risk case.

    5. After the assessment

    A Go moves to Solutioning with its value case. Carry the baseline, targets, owner, check date, and a lifecycle cost ceiling that includes applicable assurance or validation. Solutioning designs inside that ceiling. "Not this one" closes the AI idea, not the quality task.

    Want to run the assessment on your project with me? That is the work I do.

    6. Where the detail lives

    The companion post, The most valuable AI call you'll make is often "not this one", holds what this page leaves out on purpose. It includes six value metrics to pick one or two from, what each bucket needs in practice, and three sourced cases and their GxP considerations.

    The adoption line in the value case is checked again after rollout, in the AI Adoption and Improvement Framework.

    7. Caveats

    The bucket is not a regulatory classification. It does not determine regulatory applicability or set validation scope. The framework's outcomes are advisory judgments, not rulings by a regulator. The example is made up.

    Followed through, the reasoning comes down to one line:

    use fixed rules where they fit, generative AI for bounded language work that can be checked, and people for decisions that matter.