Most articles about custom AI development tell you how to pick a vendor. This one is about the work itself: what happens between “we think AI could fix this” and a system your staff use every day, what you should hold in your hands at the end of each stage, and what makes the bill bigger or smaller.
It is written from the builder’s side. AIML Marketplace is run by Empirical Edge, a custom software firm that also takes on AI and machine learning projects, so the phases below reflect how that work is usually scoped. If you are still deciding whether to hire a firm or staff up internally, read AI agency vs. in-house developers first. If you are comparing firms, see what to evaluate before hiring an AI automation company.
First question: should this be custom at all?
Google’s own engineering guidance on machine learning opens with a rule worth taking seriously: “Don’t be afraid to launch a product without machine learning.” A good builder will tell you when you do not need one.
Buy an existing product instead of commissioning a build when:
- The task is common. Meeting notes, email drafting, transcription, chat support on public FAQs, image generation and SEO writing are all well served by off-the-shelf tools. Browse the AI tools directory before you scope anything.
- Your process is standard for your industry. If dozens of companies do it the same way, someone probably sells it, often as an industry-specific product.
- You have no usable data yet. A model trained or tuned on your data needs that data to exist, be accessible and be reasonably clean.
- You cannot name an owner or a metric. If nobody will own the system after launch and you cannot say which number it should move, a build will drift.
Custom work starts to make sense when the job runs through your own systems and rules: documents in your formats, decisions that follow your policies, data that cannot leave your environment, or a workflow that spans your CRM, ERP and internal tools in a way no vendor’s roadmap will cover. Connecting AI to existing software is its own discipline, covered in AI integration services.
The four phases at a glance
Almost every well-run custom AI project moves through the same four stages, each with a decision point at the end. The point of staging is that you can stop cheaply. You should never be committed to the full production budget before the evidence says the approach works.
| Phase | Typical deliverables | What the buyer must supply | Go/no-go question |
|---|---|---|---|
| 1. Discovery and data audit | Problem statement with a success metric; data inventory and quality report; risk and compliance notes; recommended approach (including “buy, don’t build” if that is the answer); estimate for the proof of concept | A business owner; access to sample data; the people who do the work today; known security and regulatory constraints | Is there enough usable data, and a measurable target, to justify testing? |
| 2. Proof of concept | Working prototype on your real data (often a notebook or script, not an app); evaluation set and results against the agreed metric; list of failure cases; revised estimate for the MVP | A representative, agreed data sample; subject-matter experts to label or judge outputs; fast answers to edge-case questions | Did the approach hit the target on data it has not seen, at a run cost we can live with? |
| 3. MVP | A usable application for a small group of real users; one or two key integrations; access control and logging; human review steps where errors are costly; user feedback and accuracy report | Pilot users with time set aside; test environments and credentials for connected systems; a decision-maker for scope changes | Do users rely on it, and does it move the metric in real work, not just in testing? |
| 4. Production and MLOps | Hardened deployment; monitoring for accuracy, drift, cost and uptime; retraining or prompt-update process; runbooks and documentation; handover of code, models and infrastructure | An internal owner; a support agreement; IT and security sign-off; a budget line for ongoing running costs | Who owns this from now on, and what triggers a fix, a retrain or a shutdown? |
Phase 1: Discovery and data audit
Discovery is where most projects are won or lost, and it is the phase buyers most often want to skip. The output is not a slide deck. It is a short written document that says what problem is being solved, how success will be measured, what data exists, what condition it is in, and what the builder recommends.
The data audit carries the most weight. Gartner predicts that through 2026, organizations will abandon 60% of AI projects unsupported by AI-ready data. In practice, an audit answers plain questions. Where does the data live? Who can grant access? How much of it is labeled, or could be labeled by your staff? How much is duplicated, missing or out of date? Does any of it fall under HIPAA, financial regulations or a customer contract that limits where it can be processed?
A legitimate outcome of discovery is “don’t build this.” If the audit shows a SaaS product would cover 90% of the need, or that the data is not ready, you have spent a small amount to avoid a large mistake.
Phase 2: Proof of concept
A proof of concept answers one question: can this approach do this task well enough on your data? It is deliberately narrow: to automate invoice processing, it tests field extraction on a fixed sample of your real invoices, with no login screen, ERP connection or dashboard.
What you should receive:
- An evaluation set you agreed on in advance, kept separate from anything used to build or tune the system.
- Results against your metric, for example accuracy per field, or the share of answers your experts rated correct.
- The failure cases, with examples. These tell you more than the headline number.
- An estimated run cost per document, query or prediction at your expected volume.
- A revised MVP estimate based on what was learned.
This is the stage where many projects stop, and that is often the right result. Gartner predicted that at least 30% of generative AI projects would be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs and unclear business value. A proof of concept that surfaces those problems early has done its job.
Phase 3: MVP
The MVP turns a working model into something real people use in their daily work. This is where the engineering share of the budget grows: a user interface or a hook into an existing one, authentication, logging, one or two integrations with the systems the work already runs through, and error handling for the cases the model gets wrong.
Keep the pilot group small and the scope fixed. The question is no longer “does the model work?” but “does the work get faster, cheaper or more accurate when people use it?” Where a wrong output is expensive, such as a payment approval, a clinical note or a hiring decision, build a human review step into the MVP rather than adding it later. The design choices behind that are covered in human-in-the-loop AI.
By the end of the MVP you should have usage data, a measured effect on your metric, a list of the changes users asked for, and a production estimate that includes running costs.
Phase 4: Production and MLOps
Going live is not the end of spending. A Google research paper on hidden technical debt in machine learning systems warned that quick wins in ML do not come for free; the maintenance cost arrives later. Models degrade as your data, products and customers change. Hosted model APIs change versions. Business rules change.
Production deliverables should include:
- monitoring for accuracy, input drift, latency, errors and spend, with alerts that reach a named person;
- a documented process for retraining, re-tuning or updating prompts, and for testing changes before release;
- runbooks for common failures and a rollback plan;
- handover of source code, model artifacts, evaluation sets, pipelines and infrastructure configuration, as your contract specifies;
- a support agreement with response times.
The NIST AI Risk Management Framework, which is voluntary, organizes this kind of ongoing oversight into four functions: govern, map, measure and manage. It is a useful checklist when you ask a builder how the system will be watched after launch.
What drives cost and timeline
Two projects that sound alike can differ several times over in cost. The drivers are fairly predictable:
- Data readiness. Clean, accessible, labeled data is the biggest single saving. Collecting, cleaning and labeling data is slow and often needs your own experts’ time.
- Approach. Calling a hosted foundation model with your documents as context is usually faster to build than fine-tuning a model, which is usually faster than training a predictive model from scratch. Each has different run costs and data-privacy implications. The comparison of OpenAI, Google and Anthropic covers the model-provider side.
- Accuracy bar. Getting from “usually right” to “right often enough to skip review” is where effort climbs steeply.
- Number and quality of integrations. A modern system with a documented API is quick to connect. An old ERP with no API, or a vendor that charges for API access, is not.
- Compliance. HIPAA, financial regulation, data residency and audit logging add design, review and testing time.
- Buyer availability. Projects slow down most often while waiting for data access, credentials or expert review, not while waiting for code.
- Run cost and support. Hosting, API usage, monitoring and retraining are recurring. Ask for them as a separate monthly estimate.
How to use this as a buyer
Ask any builder to price and schedule the four phases separately, with the deliverables and the go/no-go question for each written into the statement of work. Make sure every phase ends with something you keep even if you stop: the data audit, the evaluation set, the prototype code. A “yes” should rest on evidence, not a sales demo.
If you are an executive setting the budget and guardrails for a first AI initiative, AI for CEOs covers that side. When you are ready to talk to firms that do this work, you can compare them in the AI/ML development provider directory.
Frequently Asked Questions
It depends mostly on how ready your data is and how narrow the question is. A proof of concept should test one thing, such as whether a model can extract the right fields from your invoices at an acceptable accuracy, on a fixed sample of your real data. If the data has to be collected, labeled or cleaned first, that work comes before the clock on the proof of concept really starts, so ask for the two to be scheduled and priced separately.
That is set by your contract, not by convention, so settle it before discovery ends. Ask specifically about source code, trained model weights, prompts and evaluation sets, data pipelines and infrastructure templates. If the system relies on a hosted model API, you will own your application and data but not the underlying model, so also check the model provider’s data-use terms for the product tier you are paying for.
Sometimes, if the task is well understood and you already know a similar approach works on data like yours. The risk is that you pay for user interfaces, integrations and access controls around a model that turns out not to be accurate enough. A short, cheap proof of concept is usually the least expensive way to find that out, which is why many projects that stall do so right after this stage.
Expect recurring costs for hosting or model API usage, monitoring, periodic retraining or prompt updates, and someone who owns the system day to day. Those costs scale with usage and with how quickly your data or business rules change. Ask any builder for a run-cost estimate at your expected volume before you approve the MVP, not after go-live.
Written by: AIML Marketplace Team
