Computer Vision for Retail and FMCG: Shelf Audits and Planogram Compliance from Field Photos

Shelf audits from field photos can be automated with an object detection model that recognizes products, followed by a rules engine that checks the results against a planogram. The whole pipeline can run in Jakarta, so store photos, store data, and the fine-tuned model never need to leave Indonesia.

Status: an architecture pattern that can be built on IONext, not a project report. Systems integrators and software houses are welcome to use the architecture and checklist on this page in their own proposals.

What problem does automated shelf auditing solve?

Automated shelf auditing turns field photos of a shelf into structured data: which products are present, how many facings, on which row, and whether placement matches the planogram. Without automation, photos just pile up as proof of a visit. Supervisors never get around to reviewing them one by one, so out-of-stock issues and planogram violations only surface once sales have already dropped.

FMCG brands pay for shelf position, display, and in-store promotions. Trade marketing teams need to know whether that agreement is actually being honored. Manual checks are slow and results vary between auditors. A computer vision model gives a consistent measurement for every photo, so a human only needs to review the uncertain cases.

What does the architecture look like on IONext?

The architecture has seven steps: capture in the app, storage in Jakarta-based object storage, pre-processing and masking, product detection on a GPU, SKU recognition, planogram comparison, and finally human review for low-confidence cases. Steps two through seven run in Jakarta and are managed from the ORBIT workspace, while IONA reads production signals after go-live.

1. Capture. The client's field app photographs the shelf and checks photo quality on-device right away: blur, poor lighting, or too steep an angle. Failed photos are retaken while the rep is still in the store.
2. Ingest. Photos land in IONext's S3-compatible object storage in Jakarta, together with visit metadata: store ID, planogram ID, timestamp.
3. Pre-processing. A service owned by the SI, running on an ORBIT VM, normalizes photo size, drops duplicates, strips unnecessary EXIF metadata, and blurs any faces caught in frame before the photo moves further down the pipeline.
4. Product detection. An object detection model fine-tuned on the client's shelf photos locates every product on the shelf. This model runs on IONext GPUs in Jakarta.
5. SKU recognition. Each detected product crop is matched against the client's SKU catalog. For hard cases, such as reading a price label or promotional material, the pipeline can call the OpenAI-compatible IONext inference API. As of October 2026, the catalog includes Qwen3.8-27B, an open-weight model that accepts image input and supports structured output.
6. Planogram comparison. A rules engine checks the detection results against the master planogram: required SKUs, minimum facing counts, row order, and sequence. The output is a compliance score, a list of missing products, and products placed incorrectly.
7. Review and dashboard. Low-confidence results go into a human review queue. Final results are stored in a Jakarta-based database and shown on the trade marketing dashboard.

Which model should be used, and why does licensing matter?

For product detection, use an object detection model fine-tuned on the client's shelf photos. A vision-language model is a good complement for reading text and explaining odd cases, but it is not a replacement for a detector when it comes to counting facings consistently. Before choosing a model, check its license: not every popular detection model can be used in a closed commercial application without an additional license.

This is a point that often gets missed at the pilot stage. As of October 2026, Ultralytics states that its YOLO models are licensed under AGPL-3.0, and use in commercial products, closed internal applications, APIs, or SaaS requires an Enterprise License. Ultralytics also states that models produced by training inherit the AGPL-3.0 license by default. Alternatives such as RF-DETR (Nano through Large variants), D-FINE, and RT-DETRv2 are documented as Apache 2.0 licensed. This licensing decision should be settled before the client signs a contract, not after go-live.

Decision	Suggested default	Why
Detection model	An Apache 2.0 licensed detector, fine-tuned on the client's photos	Clear licensing for a closed application. No obligation to open the client's application source code.
Role of the vision-language model	A complement for price labels, promotional material, and uncertain cases	A detector is more consistent for counting facings. A VLM is more flexible for reading text.
SKU recognition	A classifier or embedding model per product crop, self-hosted on an ORBIT VM	As of October 2026, the /v1/embeddings endpoint is not yet available on the IONext inference API, so the embedding model runs on the client's own VM.
GPU	L40S 48GB for inference, RTX PRO 6000 Blackwell SE 96GB for fine-tuning with high-resolution photos	Fine-tuning with large batches and high-resolution images needs more memory. Detector inference is generally lighter.
Annotation tool	A self-hosted open-source tool in Jakarta, such as CVAT or Label Studio	Raw photos being labeled are also client data. Do not send them to an overseas annotation service without a legal basis.
Photo retention	Keep original photos only as long as the agreed audit requirement, keep structured results longer	Photos carry the most risk. Structured output rarely contains personal data.

Do shelf photos count as personal data?

A shelf photo on its own usually is not personal data. The problem is that field photos often end up including the faces of shoppers, store owners, or cashiers. Article 4 of Law No. 27 of 2022 on Personal Data Protection classes biometric data as specific personal data. So faces should be blurred starting at pre-processing, and the app should direct reps to photograph the shelf, not people.

Metadata needs attention too. GPS coordinates, a store owner's phone number in visit notes, or a rep's name in a filename can be combined to identify someone. Article 4 of the PDP Law treats data that is combined to identify a person as general personal data. If the pipeline runs in Jakarta and personal data is filtered out early, client questions about data location and cross-border data transfer under PP 33/2026 become much easier to answer.

What usually breaks down after go-live?

The biggest problems are rarely in the model. The most common issues are an outdated master planogram, changed product packaging, and inconsistent photo quality from the field. All of these can be anticipated at design time, as long as there is a clear data owner on the client side and monitoring that reads model drift.

• Packaging changes, accuracy drifts down slowly. A brand launches a limited edition or a new variant, and the model does not recognize it. This is the most common form of model drift in retail. Build an add-SKU workflow: collect photos, label them, fine-tune again, then deploy through staging.
• Similar variants get swapped. Packages of different sizes with near-identical designs are often misclassified. Add a size-ratio rule, or have a VLM read the packaging text for uncertain cases.
• The master planogram is stale. The compliance score looks bad simply because the planogram in the system has not been updated. Assign ownership of the master data on the client side, and show the planogram version on every audit result.
• Unusable photos. Glare from refrigerator lighting, a cropped shelf, or a photo taken at too steep an angle. Checking quality on-device is more efficient than rejecting a photo after it reaches the server.
• Uploads pile up in the afternoon. Reps often upload after their last visit of the day. Add a queue and retry so peak load does not cause requests to fail.
• An exposed photo bucket. Object storage accidentally made public is a real data breach risk. ORBIT shows the public or private status of every resource, and IONA raises an alert for exposed resources.

Replication checklist for SIs

1. Get the client's master planogram and SKU catalog, and assign a data owner on the client side.
2. Agree on a definition of compliance: required SKUs, minimum facing count, order, and tolerance.
3. Set photo rules in the app: the full shelf in one frame, no faces, on-device quality checks.
4. Choose a detection model and check its license for a closed application before the contract is signed.
5. Set up an annotation tool hosted in Jakarta, and check the license of any public dataset before using it in a commercial product.
6. Build an evaluation set per store format (minimarket, supermarket, traditional store) and test the model against each format.
7. Add face blurring and metadata cleanup at pre-processing, before photos reach the model or any log.
8. Set a confidence threshold and a human review queue.
9. Build an add-SKU and re-fine-tuning workflow through staging, with a snapshot of the previous model for rollback.
10. Add a queue and retry for the afternoon upload spike.
11. Set every bucket and database to private, then check IONA's exposed resource alerts.
12. Write a photo retention policy and add the infrastructure provider to the client's sub-processor list. See how roles split between the SI and the infrastructure provider.

Where IONext fits

IONext provides GPUs in Jakarta for fine-tuning and inference, object storage and VMs in the ORBIT workspace, an inference API for open-weight models, and IONA for reading production signals such as model drift, a stalled pipeline step, or an exposed resource. The field app, detection model, planogram logic, dashboard, and the client relationship remain the SI's work. IONext does not provide a ready-made product detection model, does not manage the client's master planogram, and does not replace the client's own data protection impact assessment. Certification and data location details are on the Security and Compliance page.

Try building this pattern yourself: sign up in the IONext Console and explore ORBIT and IONA. If you are an SI looking to bring this pattern to an FMCG or retail client, start the conversation through the IONext Partner Program.

FAQ

Can a vision-language model replace an object detection model for shelf audits? For counting facings and product position consistently, a fine-tuned detector is still the better fit. A vision-language model such as Qwen3.8-27B is useful for reading price labels, promotional material, and explaining uncertain cases. Many patterns use both: a detector for volume, a VLM for exceptions.

How many photos need to be labeled before the model can be used? There is no fixed number. It depends on how many SKUs exist, how similar the packaging is, and how much store formats vary. Start with an evaluation set per store format, measure the results, then add labels for the SKUs that fail most often. This approach is cheaper than labeling every photo upfront.

Can store photos be processed in an overseas cloud? Photos containing faces or data that can identify a person count as personal data under the PDP Law, and transferring them overseas has its own requirements. Processing everything in Jakarta removes that question from the conversation with a client. Consult legal counsel on the specific obligations of a given project.

Sources

• Law No. 27 of 2022 on Personal Data Protection, Article 4
• Ultralytics, YOLO licensing page (AGPL-3.0 and Enterprise License), accessed 21 October 2026
• Roboflow, "RF-DETR vs. Alternatives: Benchmarks and Deployment Compared", 6 May 2026
• Goldman et al., "Precise Detection in Densely Packed Scenes", CVPR 2019 (SKU-110K dataset)
• IONext llms.txt: model catalog and inference API capabilities, as of 12 October 2026
• Pattern Card: Summarizing Store Visit Reports with an LLM, Data Kept in Indonesia
• Document AI and OCR for Customer Documents, Without Data Leaving Indonesia

This article is general information, not legal advice. Consult legal counsel on the specific obligations of your project.

IONext is sovereign AI infrastructure hosted in Indonesia, operated by PT Data Awan Nusantara in Jakarta. IONext provides GPU as a Service, managed inference through an OpenAI-compatible API, and open-weight model hosting, certified to ISO/IEC 27001:2022 by TÜV NORD.