Buying a Small Office AI Appliance: Size It Around the Work

The appliance demo works beautifully for one partner asking a short question about a clean PDF. The buying decision changes the next morning. Two colleagues open long client files while the document scanner starts an indexing batch, and the first answer takes long enough that one user returns to an unapproved web tool.

The hardware may be functioning exactly as specified. The office simply bought against a model name and memory figure instead of the work people needed to complete.

A small office AI appliance should be sized from measured documents, peak concurrency, response targets and recovery time. Hardware specifications help explain a result. The purchase earns its place only when the complete workflow passes an office acceptance test.

A focused AI development engagement can turn representative work into that test before the purchase order. The broader economics of owning inference hardware appear in running a local inference cluster. This article stays with a smaller decision: choosing one office device that fits the work and can be recovered when it fails.

---

Capture the Workload Before Looking at Devices

Start with a proposed two-week sample of normal work. Record what people ask the assistant to do and what the documents actually look like. A sizing sheet should include typical and maximum input length, expected output length, pages per document, OCR volume, languages, retrieval depth, simultaneous users and peak batch jobs.

Separate interactive work from background work. A lawyer waiting for a cited clause has a different latency requirement from an overnight indexing job. A fiduciary reviewing one payroll exception has a different memory pattern from a team comparing a year of bank statements. The same appliance may handle one workload comfortably and queue the other.

Write office-specific targets for:

• Time to the first visible response and time to a complete answer
• Maximum simultaneous users during the busiest hour
• Largest representative document set and daily OCR load
• Minimum citation or extraction quality that a reviewer will accept
• Recovery time after the device or a dependent service becomes unavailable

These are acceptance requirements rather than industry benchmarks. A partner may accept a 20-second wait for a careful contract comparison and reject the same delay for short document search. The number matters only in relation to the work.

Include German, French, Italian and English samples where the office serves those languages. Count scans, tables and handwriting. Clean digital PDFs create an optimistic picture of memory, retrieval quality and review time.

Build a Memory Budget for the Whole Stack

Model weights are only the first line of the memory budget. Add the runtime buffers, context cache for active sessions, embedding or reranking service, OCR process, operating system, document index and application. Leave working space for updates and temporary files.

Quantisation reduces the model’s memory footprint by storing weights at lower precision. The current llama.cpp documentation describes quantisation options, multiple hardware backends and hybrid inference across CPU and GPU. These techniques expand the range of models that can run on a device.

Technical fit and useful fit are separate tests. Moving part of a model or its workload into system memory may let it load, while response time grows beyond the office target. Longer context and more simultaneous sessions also expand active memory use. The candidate device therefore has to be measured with the chosen model, quantisation, runtime and real peak workload together.

Use a memory table during the proof:

Component	Measured at idle	Measured at peak	Required headroom
Model and runtime	Record on candidate	Record on candidate	Defined by update plan
Context for active sessions	Record at target length	Record at peak users	One slow-request margin
Retrieval, OCR and index	Record during search	Record while indexing	Largest batch margin
Operating system and application	Record after restart	Record during full test	Stable-service margin

The office should set the required headroom from its change and recovery plan. A universal percentage would hide differences between runtimes and workloads.

Choose a Hardware Class After the Budget

For a single-device office trial, compare three broad classes. A CPU-focused server can suit smaller models, embeddings, OCR and low-concurrency drafting. A unified-memory desktop can make a larger shared memory pool available to the model and application. A workstation with a dedicated GPU can suit workloads that fit in VRAM and benefit from its supported acceleration stack.

Each class has a different constraint. CPU-only inference may miss interactive latency targets. Unified memory is shared across the model, operating system and index. Dedicated VRAM sets the capacity boundary for GPU-resident workloads. Hybrid offload can extend technical fit into system memory, although latency and concurrency may suffer. The complete workstation also needs appropriate power and cooling.

For a concrete specification, NVIDIA lists the GeForce RTX 5090 with 32 GB of GDDR7 memory and 575 W total graphics power, together with a 1,000 W required system power in its reference details. Add-in-card designs can vary. Those figures describe one component class. They do not predict model quality, simultaneous users or the complete system’s measured electricity use.

Storage and operations often decide the better purchase. The device needs room for the current model, a previous approved version, indexes, backups or snapshots as designed, logs and temporary update files. Check warranty, replacement lead time, driver support and the administrator’s familiarity with the stack.

Avoid buying the largest memory figure available as insurance. Extra capacity has value only if the acceptance workload uses it or the approved roadmap requires it.

Measure the Complete User Experience

Run a reproducible model benchmark first, then run the same workload through the actual interface. The llama-bench documentation explains how the tool reports prompt-processing and generation measurements across repetitions. Its results exclude tokenisation and sampling time, so they should not be presented as end-to-end user latency.

For each run, record the model file, quantisation, runtime commit, context length, prompt, hardware, operating system and temperature settings. Then add the time spent on authentication, retrieval, reranking, OCR, queueing and interface rendering in the real application.

Test one user, two users and the expected peak number at the same context and document set. Mix in the background work that occurs during office hours. Record:

• Queue time, time to first token and time to complete answer
• Median and slow-request performance against the written target
• CPU, GPU and system memory at peak
• Whole-system power, temperature, throttling and errors

Averages can hide the request that sends users back to an unmanaged service. Capture p50 and p95 results for the office’s test set, while labelling both as measurements of that configuration. They are not promises for another model or workload.

Use an Acceptance Set That Tests Quality Too

Speed has no value when the answer cites the wrong page or extracts the wrong amount. Build a proposed acceptance set of 50 to 100 representative prompts with expected evidence. That range is a practical office test size for the proof; it is not a universal sufficiency threshold.

Include common tasks, difficult scans and questions whose answers are absent. Score citation correctness, extraction accuracy, abstention and reviewer acceptability by task. Keep the result by language and file type so a good overall average cannot hide a weak French scan workflow.

The same set should include concurrent use. A configuration passes only when quality and latency remain acceptable at the expected peak. Repeat it after any change to the model, quantisation, runtime, prompt, parser or retrieval settings.

Pair the quality set with operational checks on an isolated test instance using synthetic data:

1. Stop the identity service and confirm the application refuses protected retrieval.
2. Fill a test staging volume and verify alerting and safe refusal.
3. Interrupt network access, restart the device and restore from a test backup.
4. Switch to a new model, run the set and confirm that the previous approved model remains available until acceptance.

These are controlled tests with prepared recovery steps. Power or storage failures should never be induced on a live appliance holding production data.

Design Recovery Before the Device Arrives

A single office appliance is a single operational dependency. Measure restart time, backup restore time, index rebuild time and the behaviour of the application when storage or identity is unavailable. Decide which records and derived data must return together for the system to be usable.

Keep the previous approved model and configuration available until an update passes the acceptance set. Document who applies operating-system and runtime patches, who watches storage and logs, and who decides that a failed service can return to use.

The continuity route should match the sensitivity of the work. A manual workflow in the existing document, accounting or practice-management system is usually the simplest option. It asks staff to work more slowly without moving data to a new destination.

An API fallback is appropriate only when that provider, contract and processing path have been explicitly approved for the same data class. Automatic public-cloud failover can silently violate the reason the office chose local execution. Keep it disabled unless the approval is deliberate and tested.

Include recovery in the supplier demonstration. Restoring the model while losing the permission metadata or audit history is an incomplete recovery.

Calculate Ownership Cost From Measurements

Build a three-year ownership model with the same care as the workload test. Include hardware, encrypted storage, backup, UPS, network changes, installation, model and runtime integration, monitoring, support, replacement risk, staff training and professional review time.

Measure whole-system power at the wall during the acceptance workload. A GPU’s rated power cannot provide the workstation’s annual energy use by itself. The Swiss Federal Electricity Commission publishes tariff data by commune and customer category, so use the office’s applicable business tariff rather than a national guess.

For an illustrative calculation, a measured device averaging 0.65 kW while serving for 8 hours on each of 220 working days uses 0.65 × 8 × 220 = 1,144 kWh. At an illustrative tariff of CHF 0.24 per kWh, the serving-energy term is 1,144 × CHF 0.24 = CHF 274.56 per year. This illustrative amount excludes idle consumption, cooling and backup loads. Capital, support and administrator costs must come from the dated quote and the office’s own staffing assumptions.

Compare a managed private service at the same document volume, quality target and data classification. Add internal administration to both choices. Local hardware may remove a per-token fee while adding patching and recovery work. A managed service may reduce operations while introducing provider, contract and jurisdiction questions.

The custom versus platform TCO model provides a wider cost framework. For this purchase, the measured workload and named operating owner matter more than a generic hardware range.

Buy the Smallest Configuration That Passes

The appliance decision should end with a result sheet rather than a list of impressive specifications. Record which device and software stack passed the office’s quality, concurrency, latency and recovery targets. Note the headroom at peak and the tasks that still require the manual route.

This makes the value of local inference practical. Staff receive a useful assistant for approved sensitive work, and the office knows how it behaves when several people use it or a component fails. The device is sized to a real process instead of a model leaderboard.

Choose the smallest configuration that passes with enough measured headroom for the planned workload. Re-run the same tests after meaningful changes. That discipline keeps the appliance useful long after the purchase decision and gives the office evidence for the next upgrade instead of another guess.