Property Finder: 96.8% of its eval labels came from the model it replaced

On 27 October 2026 AWS published a post, co-authored with three Property Finder engineers, on how the Middle East and North Africa (MENA) property marketplace changed the vision model that checks its listing photos. Property Finder reports that 96.8 percent of its evaluation labels had been copied from the existing model's predictions rather than checked by a person. This is vendor-published: AWS's Generative AI Innovation Center ran the engagement and the post promotes Amazon Bedrock. Every number below is the authors'.

The evaluation was grading the incumbent's homework

Property Finder scores tens of thousands of listing images a day on 11 quality attributes, including watermark detection, room classification, furnishing status and photo quality. Wanting a cheaper model, the team measured candidates against its existing labels and found that almost all of them (96.8 percent) were the incumbent vision language model's own output. The post calls this a circular evaluation: any alternative was judged by how closely it matched the incumbent, not by whether it was right.

The labels were also wrong in places. According to the post, character-level mistakes in watermark text, including hallucinated brand names, were marked correct. Single labels were forced onto photos showing more than one room. Of the 3.2 percent of labels that humans had written, some used categories outside the specification, such as Office.

What they did instead

• One domain expert with real-estate listing-quality knowledge hand-annotated 356 UAE images, without seeing any model output. The first 256 formed a development set for tuning; 100 more, labelled after the tuning runs, were added for the final comparison. A separate Egyptian set of 230 images was a cross-regional check and is not reported.
• They moved room and photo classification from single-label to multi-label, so a hallway photo showing a bedroom through a door is scored as both.
• They scored with Micro F1, plus Macro F1 so rare classes were not hidden, per-sample intersection over union, and exact-match accuracy on the nine single-label attributes.
• They ran a prompt-optimisation loop from version 0 to version 27, with AI assistants proposing changes from failed images and, in version 27, the DSPy library's MIPROv2 optimiser restructuring the photo-classification task.

Results, and how far to trust them

The post reports that the incumbent, called proprietary model A, scored 0.55 Micro F1 on the 256-image development set. Meta's Llama 4 Maverick 17B Instruct reached 0.75 on the shared version-21 prompt and 0.81 with its tuned version-27 prompt, a 47 percent relative gain over the incumbent. On the 356-image evaluation set the final prompt scored 0.80 against 0.56. By the authors' account, Llama matched or beat the incumbent on all 11 attributes. It scored 90 percent or higher exact-match on six of nine single-label attributes, including 100 percent on rotation and 94 percent on watermark detection, but only 65 percent on room size and 55 percent on watermark text extraction.

The authors list their own caveats. Both models' prompts were tuned on the development images, which sit inside the evaluation set, so the scores are described as optimistic. The incumbent got one tuning round (version 22) and Llama got several (versions 24 to 27) plus Property Finder's own edge-case rules, so Llama had more tuning effort behind it. A set of 356 images, the post says, detects only moderate differences, and rare classes have too few examples to show smaller ones.

Cost and speed

The post estimates $2.35 per 1,000 images for Llama on Bedrock, 86 percent below the estimate for Property Finder's previous setup. Those are estimates as of December 2025 in the US East (Ohio) Region, assuming 9,000 input and 200 output tokens per image from an earlier two-call analysis; the production single call uses about 7,500 to 8,000 input tokens, so the post calls it an upper estimate. The 86 percent compares against a setup using the version-21 prompt, while the accuracy comparison used version 22, so its cost may differ. Property Finder separately reports a 40 to 60 percent drop in inference latency in its own monitoring. The post does not name the incumbent model or give its price.

What did not work, and what the post leaves out

Splitting the watermark attributes into a separate API call brought no meaningful accuracy gain and made boxed-image detection fall sharply, because the models mistook watermark overlays for borders. Generic prompts failed on every model. The post gives no human-review rate, no error cost for a wrongly flagged listing, and no independent replication. The authors note that the pipeline produces no calibrated confidence score, and suggest model disagreement as a signal for which images to send to a person, to be validated on a development set first.

What to check in your own evaluation

The portable lesson is the audit. The post tells readers to find out whether their evaluation labels came from model predictions, to have domain experts label a few hundred representative samples independently, and to keep a test split that is never used for tuning. If your current model produced your answer key, a cheaper candidate may look worse than it is.