Friday, May 1, 2026

Power Platform + AI Builder Complete Guide

 

Power Platform + AI Builder — Complete Guide

AI Models · Document Processing · Prompt Builder · GPT · Object Detection · Integration · Scenarios · Cheat Sheet


Table of Contents

  1. Core Concepts — Basics
  2. AI Model Types — Deep Dive
  3. Document Processing — Prebuilt & Custom
  4. Prompt Builder & GPT Capabilities
  5. Integration with Power Platform
  6. Governance, Licensing & Best Practices
  7. Scenario-Based Questions
  8. Cheat Sheet — Quick Reference

1. Core Concepts — Basics

What is AI Builder and how does it fit in the Power Platform?

AI Builder is a Microsoft Power Platform capability enabling makers to add AI intelligence to their apps and automations without data science expertise. It provides prebuilt and custom AI models usable directly in Power Apps, Power Automate, and Copilot Studio.

AI Builder's role in the Power Platform:

  1. Power Apps: embed AI controls — scan receipts, analyse sentiment, detect objects from camera
  2. Power Automate: use AI models as flow actions — extract invoice data, classify documents, run GPT prompts
  3. Copilot Studio: use Prompt Actions to call AI Builder GPT prompts from bot conversations
  4. SharePoint Syntex: AI Builder document processing models underpin Syntex auto-classification

Key positioning: AI Builder democratises AI — it makes document intelligence, object detection, and language AI accessible to makers without ML expertise or Azure AI setup.


What are the two categories of AI Builder models?

Prebuilt models: ready-to-use, trained by Microsoft. No training required — use immediately.

Prebuilt Model Extracts/Detects
Invoice processing Vendor, invoice number, dates, totals, line items
Receipt processing Merchant, date, total, line items, payment method
Business card reader Name, title, company, email, phone, address
Identity document reader Name, DOB, nationality, document number, expiry
Contract processing Parties, dates, obligations, key clauses
Sentiment analysis Positive/Negative/Neutral/Mixed + confidence scores
Language detection Detected language + confidence
Text recognition (OCR) All text from images and PDFs

Custom models: trained on your own data for specific business needs.

Custom Model Purpose
Document processing Extract fields from your unique document layouts
Object detection Detect your specific objects in images
Category classification Classify text into your custom categories
Text classification Multi-label text classification

Tip: Always start with prebuilt models. Only build custom models when prebuilt models don't cover your specific document layout, object type, or classification category.


What is the AI Builder model lifecycle?

Custom model lifecycle:
1. Create   → define model type and target fields/categories
2. Train    → provide labelled training examples
             → AI Builder trains on your data (minutes to hours)
3. Evaluate → review performance metrics (accuracy, F1, confidence)
             → if insufficient: add more examples → retrain
4. Publish  → make model available in Power Apps and Power Automate
5. Use      → call via AI Builder connector in apps and flows
6. Retrain  → when new document types or edge cases emerge

Performance metrics:
Precision → of predicted positives, how many were correct
Recall    → of actual positives, how many were correctly predicted
F1 Score  → harmonic mean of precision and recall (target: > 0.7 for production)
Confidence → per-prediction score (0-1) — use threshold to filter low-confidence results

What is AI Builder licensing and how are credits consumed?

AI Builder uses a capacity-based model measured in AI Builder credits. Credits are consumed each time a model processes a request.

Model Approximate Credit Cost
Document processing ~2 credits per page
Prompt (GPT) Scales with token usage
Sentiment analysis ~1 credit per call
OCR / Text recognition ~1 credit per page
Object detection ~1 credit per image

Licence sources:

  • Included allocation with some Power Apps/Automate plans
  • AI Builder add-on capacity pack
  • Trial credits for new environments

Warning: AI Builder credits are shared across the environment. High-volume automations can exhaust credits quickly. Monitor usage: Power Platform Admin Centre → AI Builder → Credits.


2. AI Model Types — Deep Dive

What is the Text Recognition (OCR) model?

The Text Recognition model (prebuilt) extracts all text from images and PDF documents using Optical Character Recognition, returning text content and bounding box coordinates.

Use cases:
→ Extract text from scanned documents
→ Read handwritten notes or forms
→ Process images with printed text (signs, labels, packaging)
→ Convert scanned PDFs to searchable text

Power Automate output:
{
  "pages": [{
    "lines": [{
      "text": "Invoice Number: INV-2025-001",
      "words": [
        { "text": "Invoice" },
        { "text": "Number:" },
        { "text": "INV-2025-001" }
      ]
    }]
  }]
}

Tip: Text Recognition is a foundational model — often the first step in document processing pipelines where specialised prebuilt models don't apply.


What is the Sentiment Analysis model?

The Sentiment Analysis model (prebuilt) analyses text and returns sentiment classification at document and sentence level.

Sentiment output:
Overall sentiment: Positive / Negative / Neutral / Mixed
Confidence scores: { positive: 0.92, neutral: 0.05, negative: 0.03 }

Sentence-level:
"The product is excellent."   → Positive (0.97)
"Delivery was very slow."     → Negative (0.89)
"The packaging was fine."     → Neutral (0.71)
Document overall: Mixed

Use cases in Power Platform:
→ Route negative customer emails to priority queue
→ Analyse survey responses in Power Apps
→ Flag negative Omnichannel conversations to supervisors
→ Build sentiment trend dashboards in Power BI

What is the Object Detection model?

Object Detection identifies and locates specific objects in images, returning names and bounding box coordinates.

Training a custom Object Detection model:
1. Define object types: "Scratch", "Dent", "Missing Label", "Correct"
2. Upload training images: minimum 15 per object type
3. Tag objects: draw bounding boxes in each image
4. Train: AI Builder learns layout pattern
5. Evaluate: mAP (mean Average Precision) score (target: > 0.75)
6. Publish and use

Use cases:
→ Quality control: detect defects in manufacturing photos
→ Retail inventory: count products on shelves
→ Safety inspection: detect missing PPE in site photos
→ Insurance: detect vehicle damage in claim photos

In Power Apps:
Control: AI Builder Object Detector
→ Camera input → returns detected objects + bounding boxes
→ Display bounding box overlays on the captured image

What is the Category Classification model?

Category Classification (custom) classifies text into categories using your own training examples. Supports single-label and multi-label classification.

Training:
1. Define categories: "Complaint", "Compliment", "Query", "Refund Request"
2. Provide training examples: minimum 10 text examples per category
3. Train and evaluate (F1 target: > 0.7)
4. Publish

Multi-label example:
Text: "The product broke after 2 days and I want a refund"
Labels: ["Complaint", "Refund Request"]  ← both apply simultaneously

In Power Automate:
Action: AI Builder → Classify text into categories
→ Input: email body
→ Output: topClass (string), allClasses (array with confidence scores)
→ Use topClass to route flow to the correct team or queue

3. Document Processing — Prebuilt & Custom

What do the prebuilt document processing models extract?

Invoice processing extracts:

{
  "VendorName": "Contoso Supplies Ltd",
  "InvoiceId": "INV-2025-4421",
  "InvoiceDate": "2025-03-15",
  "DueDate": "2025-04-15",
  "InvoiceTotal": { "amount": 5250.00, "currencyCode": "GBP" },
  "LineItems": [
    { "Description": "Office Supplies", "Quantity": 10,
      "UnitPrice": 450.00, "Amount": 4500.00 },
    { "Description": "Delivery", "Quantity": 1,
      "UnitPrice": 750.00, "Amount": 750.00 }
  ]
}

Other prebuilt models:

  • Receipt: merchant name, date, total, subtotal, tax, line items, payment method
  • Business card: name, job title, company, email, phone, address, website
  • Identity document: name, DOB, nationality, document number, expiry date
  • Contract: parties, effective date, expiry date, obligations, governing law

How do you train a custom Document Processing model?

Steps:
1. Create model in AI Builder → select "Document processing"
2. Define fields to extract:
   e.g., PolicyNumber, CustomerName, CoverageAmount, EffectiveDate, ExpiryDate
3. Create document collections:
   → Upload 5+ sample documents per layout variation
   → Use real representative production documents
4. Tag fields:
   → Draw bounding boxes around each field value in each sample
   → Tag consistently across all samples
5. Train the model
6. Evaluate: accuracy per field (target: > 90%)
7. Publish and use in flows

Multiple layout support:
One model handles multiple document layouts:
  Auto Policy / Home Policy / Life Policy
  Model auto-detects layout type from the document

Tip: 5 training documents is the minimum — but 15-20 samples per layout dramatically improves accuracy. Always test with documents that were NOT in the training set.


How do you handle low-confidence extractions?

Every extracted field includes a confidence score (0–1). Low confidence = model is uncertain.

Confidence-based routing pattern:

AI Builder extracts:
  InvoiceNumber: "INV-001" (confidence: 0.95) ✓
  Amount:        "£5,250"  (confidence: 0.42) ← LOW
  VendorName:    "Contoso" (confidence: 0.88) ✓

Condition: IF Amount confidence < 0.80
  True  → Route to human review queue
          Send Teams notification: "Manual review needed: INV-001"
          Create Planner task for AP team
  False → Automate processing fully

Confidence thresholds (recommended):
> 0.90 → High confidence: automate fully
0.70–0.90 → Medium: automate with audit logging
< 0.70 → Low: route for human review
0.00 → Field not found in document

Critical: Never automate 100% of documents without a human review path for low-confidence extractions. Data errors in automated financial processes compound quickly.


4. Prompt Builder & GPT Capabilities

What is Prompt Builder in AI Builder?

Prompt Builder is a low-code interface for creating, testing, and publishing GPT-based AI prompts usable in Power Automate, Power Apps, and Copilot Studio. Uses Azure OpenAI under the hood.

Capabilities:

  • Text generation: email drafts, summaries, descriptions, meeting notes
  • Text transformation: translate, rewrite, reformat, extract key points
  • Classification: classify text using natural language instructions
  • Information extraction: extract specific information from unstructured text
  • Question answering: answer questions based on provided context
Prompt template example:
─────────────────────────────────────────────
System prompt:
You are a professional customer service agent for Contoso.
Respond in a friendly, concise tone. Maximum 3 paragraphs.
Do not make up specific dates, prices, or policy details.

User prompt:
Customer email: [EmailBody]
Customer name: [CustomerName]
Customer tier: [CustomerTier]

Task: Draft a professional response to this customer email.
If Gold tier, offer priority callback within 2 hours.
Reference the customer by name.
─────────────────────────────────────────────

How do you use Prompt Builder in Power Automate?

Flow: Auto-summarise customer feedback

Trigger: SharePoint list item created (feedback form response)

Action 1: Get form response
  → CustomerName, FeedbackText, ProductName, Rating

Action 2: AI Builder → Create text with GPT using a prompt
  → Select: "Summarise Customer Feedback" prompt
  → Dynamic inputs:
    [FeedbackText] = triggerBody()?['FeedbackText']
    [ProductName]  = triggerBody()?['ProductName']
    [Rating]       = triggerBody()?['Rating']
  → Output: generatedText

Action 3: AI Builder → Detect sentiment
  → Input: FeedbackText
  → Output: sentiment, confidenceScores

Action 4: Create Dataverse record
  → CustomerName, OriginalFeedback, AISummary (generatedText),
    Sentiment, Rating, ProductName

Action 5: Condition — IF sentiment = 'Negative' AND Rating <= 2
  → Send Teams alert to Customer Success team

Access prompt output:
@{outputs('Create_text_with_GPT')?['text']}

What is a Prompt Action in Copilot Studio?

A Prompt Action allows a Copilot Studio bot to call an AI Builder Prompt directly from a topic — without a Power Automate flow intermediary.

Copilot Studio topic flow:
User: "Summarise this text for me: [pastes paragraph]"

Question node → collect text → varInputText

Action node → AI Builder Prompt Action
  → Select: "Summarise Text" prompt
  → Input: varInputText
  → Output: varSummary

Message node → "Here is your summary:"
  → Display: {varSummary}

Use cases for Prompt Actions in bots:
→ Summarise long documents pasted by users
→ Draft responses to customer queries
→ Translate text in real time during conversations
→ Extract key points from meeting notes
→ Answer questions using provided context text

What are best practices for writing effective AI Builder prompts?

  1. Be specific and explicit: "Extract the invoice total as a number only, without currency symbol" not "get the amount"
  2. Define output format: "Return a JSON object with: summary (string), sentiment (Positive/Negative/Neutral), action_required (true/false)"
  3. Set constraints: "Maximum 3 sentences", "Only use information present in the input", "Use formal English"
  4. Provide role/context: "You are a professional legal summariser for a UK law firm"
  5. Handle edge cases: "If the input does not contain an invoice number, return 'NOT_FOUND'"
  6. Test with varied inputs: empty input, very long text, different languages, malformed input
  7. Version prompts in source control: prompts are code — treat them as such

Warning: Never process PII, financial details, or sensitive data through GPT prompts without reviewing your organisation's data privacy policies and Microsoft's data protection commitments.


5. Integration with Power Platform

How do you use AI Builder in Power Apps canvas apps?

// AI Builder Receipt Processor in Power Apps
// Add control: Insert → AI Builder → Receipt Processor

OnScan formula:
Set(varReceipt, 'Receipt processor 1'.VisionResponse);
Set(varMerchant, varReceipt.MerchantName.value);
Set(varAmount,   varReceipt.Total.value);
Set(varDate,     varReceipt.TransactionDate.value);

// Auto-fill form fields
UpdateContext({
  txtMerchant: varMerchant,
  txtAmount:   varAmount,
  txtDate:     varDate
})

// Confidence check — warn if low
If(varReceipt.Total.confidence < 0.8,
  Notify("Please verify the total amount — low confidence scan",
    NotificationType.Warning)
)

AI Builder controls available in Power Apps:

  • Business Card Reader
  • Form Processor (custom document models)
  • Object Detector
  • Text Recogniser
  • Receipt Processor

How do you build an end-to-end invoice processing automation?

Complete flow architecture:

Trigger: New PDF file in SharePoint "Invoice Inbox" library

Step 1: Get file content (SharePoint)

Step 2: AI Builder → Process and save information from invoices
  Outputs: VendorName, InvoiceId, InvoiceDate, DueDate,
           InvoiceTotal {amount, currency}, LineItems[]

Step 3: Confidence check
  IF InvoiceTotal.confidence < 0.85:
    → Move to "Requires Review" folder
    → Create SharePoint task for AP team
    → STOP automatic processing
  ELSE: Continue

Step 4: Create Dataverse Invoice record
  → Map all AI outputs to table columns
  → Attach link to original SharePoint file

Step 5: Amount-based approval routing
  IF amount > £10,000 → Finance Director approval
  IF amount > £1,000  → Finance Manager approval
  ELSE → Auto-approve

Step 6: Move processed file to "Processed" folder

Step 7: Update SharePoint Invoice register list

Error handling (Catch scope):
  → Log to Dataverse Error Log table
  → Send Teams alert to AP team with error details

What is the difference between AI Builder and Azure Document Intelligence?

AI Builder Azure Document Intelligence
Interface Low-code GUI REST API / SDK (pro-code)
Azure subscription Not required Required
Training GUI-based tagging API or Document Intelligence Studio
Pricing AI Builder credits Pay-per-page
Customisation Standard custom models Advanced: neural models, query fields
Integration Power Platform native Any application (Azure Functions, .NET)
Best for Power Platform makers High-volume, lower cost, non-PP systems

Tip: Use AI Builder when the solution is entirely within Power Platform. Use Azure Document Intelligence directly for higher volume, lower per-page cost, or integration with non-Power Platform systems.


6. Governance, Licensing & Best Practices

What are the key AI Builder governance considerations?

  1. Data privacy: AI Builder sends data to Azure AI services — ensure your data handling complies with GDPR and organisational data classification policies. Never send highly confidential or regulated data (medical records, financial secrets) without explicit policy approval.

  2. Credit monitoring: set up alerts in Power Platform Admin Centre when credit consumption exceeds thresholds. Identify top-consuming flows and apps.

  3. Model ownership: custom models should be owned by a service account, not an individual — prevents orphaned models when staff leave.

  4. Prompt governance: treat prompt templates as code — version them in source control, review them before production use, document what inputs they expect and what they produce.

  5. Confidence threshold policies: define organisation-wide confidence thresholds for automated vs human-reviewed processing. Document these policies for audit purposes.

  6. DLP policies: Power Platform DLP policies apply to AI Builder actions in flows — the AI Builder connector is a premium connector subject to tenant DLP rules.


What are the AI Builder model performance targets?

Minimum recommended production thresholds:
Document processing:    > 90% accuracy per field
Category classification: F1 score > 0.75
Object detection:       mAP > 0.75
Sentiment analysis:     Prebuilt — no training needed
Text recognition:       Prebuilt — accuracy depends on document quality

Signs a model needs retraining:
→ Confidence scores dropping on new documents
→ New document layouts not covered by training data
→ Business changes altering field names or positions
→ Increased human review queue volume

Retraining triggers:
→ Monthly review of confidence score trends
→ When > 10% of documents require manual review
→ After any significant document layout change

7. Scenario-Based Questions

Scenario: Automate expense claim processing with mobile receipt scanning.

Architecture:

  1. Power Apps mobile canvas app:

    • Camera control → employee photographs receipt
    • AI Builder Receipt Processor → auto-extracts merchant, amount, date, category
    • Form auto-fills — employee reviews and corrects if needed
    • Employee adds project code and description
    • Submit → calls Power Automate flow
  2. Power Automate flow:

    • Receive claim data from Power Apps
    • AI Builder Category Classification → flag "personal" vs "business" expenses
    • Confidence check: if receipt amount confidence < 0.8 → flag for manual review
    • Approval routing: > £50 → manager; > £500 → manager + finance
    • Store in Dataverse Expense table + attach receipt image
  3. Power BI dashboard: expense by category, employee, project, month

Tip: The Power Apps (capture) + AI Builder (extract) + Power Automate (process) + Power BI (report) stack is a complete citizen developer solution that would otherwise require a custom enterprise application.


Scenario: Build an AI-powered customer email triage system.

Flow architecture:

Trigger: New email arrives in support@contoso.com shared mailbox

Step 1: AI Builder Sentiment Analysis
  → Determines: Positive / Negative / Neutral / Mixed

Step 2: AI Builder Category Classification (custom model)
  → Categories: Billing / Technical / Refund / Complaint / General

Step 3: AI Builder Prompt (GPT)
  → Extract: account number, product, issue summary (1 sentence)

Step 4: Routing (Switch on Category):
  Complaint + Negative → Priority queue + manager Teams alert
  Refund Request       → Finance team queue
  Technical Issue      → IT support queue
  Billing Query        → Billing team queue
  General              → Standard queue

Step 5: Create D365 Customer Service case
  → Subject: [Category], Description: [AI summary]
  → Customer: matched by extracted account number
  → Priority: derived from sentiment

Step 6: AI Builder Prompt → draft personalised auto-acknowledgement
  → Includes case number and expected response time by tier

Scenario: Quality control inspection app using Object Detection.

  1. Train custom Object Detection model:

    • Objects: "Scratch", "Dent", "Missing Component", "Correct Product"
    • Collect 30+ labelled photos per object from production line
    • Target mAP > 0.75
  2. Power Apps inspection app:

    • Camera control → inspector photographs product
    • AI Builder Object Detector → returns detected defects + bounding boxes
    • App overlays bounding boxes on image with defect labels
    • Inspector reviews AI findings, adds notes, confirms pass/fail
  3. Power Automate on submit:

    • Store inspection in Dataverse (product ID, inspector, result, defects)
    • If defect detected → Teams alert to production supervisor
    • If defect rate > 5% today → alert quality manager
  4. Power BI: defect rate by product line, time of day, inspector


Scenario: Contract management system with AI Builder document intelligence.

  1. AI Builder Contract Processing model: extract parties, effective date, expiry date, contract value, key obligations, governing law

  2. SharePoint intake library: "Contract Inbox" — legal team drops incoming contracts

  3. Power Automate flow on upload:

    • AI Builder extracts all contract fields
    • Confidence check: any field < 0.85 → route to paralegal review
    • Create Dataverse Contract record with all extracted fields
    • Schedule expiry reminder: flow runs 60 days before expiry → notify contract owner
  4. AI Builder Prompt: generate plain-English summary of key obligations (5 bullet points)

  5. Power Apps contract dashboard: all contracts, filter by expiry, search by party, view AI summaries

  6. Purview retention: 7-year retention on the SharePoint contract library

Result: Complete contract lifecycle management built entirely in Power Platform — no custom code required.


8. Cheat Sheet — Quick Reference

Model Selection Guide

Need to...                                    → Use
Extract text from any image/PDF               → Text Recognition (OCR) — prebuilt
Process invoices                              → Invoice Processing — prebuilt
Process receipts/expenses                     → Receipt Processing — prebuilt
Read business cards                           → Business Card Reader — prebuilt
Analyse text sentiment                        → Sentiment Analysis — prebuilt
Detect language                               → Language Detection — prebuilt
Extract fields from YOUR document layout      → Custom Document Processing
Detect YOUR specific objects in images        → Custom Object Detection
Classify text into YOUR categories           → Custom Category Classification
Generate/transform/summarise text (GPT)       → Prompt Builder
Answer questions from context                 → Prompt Builder

Confidence Threshold Reference

Recommended automation thresholds:
> 0.90 → Automate fully — high confidence
0.70–0.90 → Automate with audit logging
< 0.70 → Route to human review queue
0.00 → Field not found — always human review

In Power Automate expression:
outputs('Process_Invoice')?['responsev2']?['predictionOutput']?['labels']?['InvoiceTotal']?['value']
outputs('Process_Invoice')?['responsev2']?['predictionOutput']?['labels']?['InvoiceTotal']?['confidence']

Custom Model Training Requirements

Document Processing:
  Minimum training documents: 5 per layout
  Recommended: 15-20 per layout
  Accuracy target: > 90% per field

Object Detection:
  Minimum images per object: 15
  Recommended: 30+ per object
  Performance target: mAP > 0.75

Category Classification:
  Minimum examples per category: 10
  Recommended: 50+ per category
  Performance target: F1 > 0.75

Retrain when:
→ > 10% of documents requiring manual review
→ New document layouts introduced
→ Business changes affecting field positions
→ Monthly confidence score review shows decline

AI Builder in Power Platform — Integration Summary

In Power Automate:
Action: AI Builder → [model name]
→ Input: file content, text, or image
→ Output: structured extraction results

In Power Apps:
Control: AI Builder [model] (from Insert menu)
→ Camera or image picker input
→ Outputs available as control properties

In Copilot Studio:
Action node → AI Builder Prompt Action
→ Select saved prompt from AI Builder
→ Pass topic variables as prompt inputs
→ Output stored in topic variable

Prompt output in Power Automate:
@{outputs('Create_text_with_GPT')?['text']}

Document processing confidence:
@{outputs('Process_document')?['responsev2']?['predictionOutput']?['labels']?['FieldName']?['confidence']}

Top 10 Tips

  1. Prebuilt before custom — always start with prebuilt models. Custom models require training data, maintenance, and retraining. Only build custom when prebuilt doesn't cover your use case.
  2. Confidence-based routing is mandatory — low-confidence extractions routed to human review. Never 100% automate without a manual fallback. This is the Responsible AI answer.
  3. AI Builder = low-code Azure AI — AI Builder is a Power Platform wrapper over Azure Document Intelligence, Azure AI Language, and Azure OpenAI. Know the underlying services for architect-level questions.
  4. Credits are shared across the environment — high-volume automations consume credits fast. Monitor usage proactively. Design flows to process only when needed (trigger conditions).
  5. Document Processing vs Text Recognition — Document Processing extracts specific named fields. Text Recognition extracts all text (OCR). Use Document Processing for structured extraction; OCR for unstructured digitisation.
  6. Prompt output format matters — specify the exact output format in the prompt (JSON, comma-separated, numbered list). Unstructured GPT output is hard to parse in flows. Always define the schema.
  7. Custom Object Detection needs 15+ images per class — below this, accuracy is unreliable. 30+ is production-ready. Ensure images are varied (different angles, lighting, backgrounds).
  8. Prompt Actions in Copilot Studio — the simplest way to add generative AI to a bot topic without a Power Automate flow. Know this pattern — it comes up in every combined Copilot Studio + AI Builder .
  9. AI Builder + D365 Customer Service — sentiment analysis routing negative emails to priority queue, category classification routing to correct team, Prompt drafting acknowledgements. This combined scenario is a favourite enterprise question.
  10. Data privacy with GPT prompts — never send PII, medical, or highly confidential data through Prompt Builder without explicit organisational approval. Always mention this in any prompt-related discussion to demonstrate governance awareness.


No comments:

Post a Comment

Featured Post

Power Platform + AI Builder Complete Guide

  Power Platform + AI Builder — Complete Guide AI Models · Document Processing · Prompt Builder · GPT · Object Detection · Integration · Sc...

Popular posts