If you are deciding whether visual search is ready for your B2B marketplace, do not start with image-model accuracy. Start with a harder product question: what does the buyer mean by this image?

A system can return products that look remarkably similar and still fail the sourcing task. The buyer may need an exact model, a compatible substitute, the item named on a label, or merely a product with the same design. Your product has to resolve that intent before ranking can become trustworthy.

Query provenance changes the job to be done

The clean camera photo is only one kind of visual query. Real uploads also include competitor-listing screenshots, social-media screengrabs, packaging, labels, and frames extracted from video. These are not interchangeable versions of a noisy product photo. Each preserves different evidence and implies a different initial hypothesis about the buyer’s goal.

Uploaded image	Useful initial hypothesis	Evidence to prioritize	Best clarification
Photo of an object in front of the buyer	Identify, replace, or find an alternative to the object	Object category, shape, components, visible attributes, and any markings	Do you need this exact item, a compatible replacement, or a similar product?
Screenshot of a competitor listing	Source the same product or compare available offers	Listing title, brand, model number, variant text, and the product crop	Which details must remain the same?
Packaging or label	Find the commercial item identified by the printed information	Brand, product name, strength, grade, size, and other identity fields	Should results match the exact labeled item or include alternatives?
Rating plate, model sticker, or specification sheet	Find an exact part or a compatible replacement	Part number, model, manufacturer, ratings, dimensions, and standards	Is an exact identifier match required, or is compatibility acceptable?
Social-media image or video frame	Find the featured object or a visual analogue	The buyer-selected object, surrounding context, and any accompanying text	Which object in the image are you looking for?

Treat those rows as hypotheses, not deterministic rules. A camera photo can still be a photograph of a label. A competitor screenshot can be inspiration rather than an exact-match request. Provenance should influence the first branch in the experience, but the buyer must be able to correct it.

Capture provenance before the search request is flattened into an embedding. Your event and query schema should preserve:

• The entry channel, such as camera, file upload, clipboard, or shared image.
• The original image and the buyer’s crop or selected object, subject to your retention and privacy rules.
• Whether the image contains substantial text, multiple objects, packaging, or a likely specification plate.
• The extracted identifiers and attributes, including the unmodified text as well as normalized candidates.
• The intent option the buyer selected, ignored, or later changed.
• The filters, reformulations, shortlisted products, inquiries, and other downstream actions.

This gives you something more valuable than a generic record of an image search: a trace of how the system interpreted the request and where that interpretation changed.

Convert the image into an explicit query contract

A text query can move from broad to narrow: a product category, a brand, a family, and then a specific model. A photograph cannot express that range. It presents one concrete item with a particular color, configuration, condition, and viewing angle, even when the buyer does not care about most of those properties.

Consider a photograph of an industrial motor. It could mean, “Find this exact model.” It could mean, “Find a compatible replacement.” It could also mean, “Find any motor with this enclosure design, subject to my voltage and mounting requirements.” The pixels do not change across those requests, but the acceptable result set changes completely.

Represent that acceptable set as a query contract. It should contain six parts:

1. Target: the object, label, or region the buyer means. This matters when an image contains a product, accessories, packaging, and background objects.
2. Intent mode: exact identity, compatible alternative, visual analogue, or category exploration.
3. Identity evidence: brand, manufacturer, product name, part number, model, stock-keeping identifier, or another category-specific key.
4. Hard constraints: attributes that every acceptable result must satisfy.
5. Soft preferences: properties that improve ranking but may be traded off, including visual resemblance when it is relevant.
6. Commercial constraints: quantity, location, supplier requirements, lead-time expectations, price boundaries, and other conditions involved in sourcing.

The interface does not need to display a six-field form. It needs to make the most consequential ambiguity cheap to resolve. After detecting a model plate, ask whether the buyer wants the exact part number or a compatible replacement. In a design-led category, offer controls for shape, color, material, or overall resemblance. If the image contains several products, ask the buyer to tap or crop the target before running retrieval.

When the buyer does not answer, keep the ambiguity visible. Separate results into lanes such as “Exact identity,” “Compatible alternatives,” and “Similar appearance” rather than blending all three into an unexplained ranking. Only show a lane when your catalog data can support its claim; a visually related item is not automatically compatible.

Text is the control surface for this contract. Its deepest role is not simply to contribute details missing from the photo. It tells the system which visible properties count and which should be ignored. It also carries constraints a camera may not reveal, including material grade, tolerances, certifications, and order quantities.

Do not force the buyer to retype text you have already extracted. Present brand, model, size, or grade as editable chips. This turns extraction errors into correctable product interactions instead of silent retrieval failures.

Route identity queries away from pure visual similarity

Some images are better understood as containers for text. A box of medicinal gel is a clear example: the relevant evidence can be the molecule, brand, product name, and strength rather than the package design. Another product can look almost identical while being the wrong result. The same product can arrive in redesigned packaging and remain correct.

Model-number stickers, rating plates, barcodes, datasheets, and listing screenshots create the same architectural requirement. Visual resemblance is not merely a weaker signal in these cases. It can point the ranking in the wrong direction.

A robust retrieval pipeline should preserve separate evidence channels until intent is known:

1. Locate the target. Detect likely products, text regions, labels, and specification plates. Preserve the full image as context while allowing the buyer to correct the selected region.
2. Route the query. Estimate whether it is appearance-led, identity-led, or hybrid. Treat this as a revisable decision, not an invisible verdict.
3. Extract evidence. Produce visual descriptors, recognized text, identifier candidates, brands, models, and category-specific attributes. Keep original and normalized forms so normalization cannot erase useful evidence.
4. Resolve entities. Match extracted evidence against canonical manufacturers, products, variants, and identifiers. Account for punctuation differences, aliases, misspellings, and common character confusions without pretending an uncertain match is exact.
5. Generate candidates in parallel. Use exact-identifier retrieval, lexical or semantic text retrieval, structured-attribute retrieval, and visual-vector retrieval as distinct candidate generators.
6. Apply the query contract. Use confirmed identifiers and hard constraints as gates. Use soft attributes and visual similarity for ranking within the eligible set.
7. Re-rank conditionally. Change the scoring logic by category and intent mode. Exact identifiers should dominate an identity query; visual evidence should carry more weight in an appearance-led query.
8. Explain or abstain. Show why a result matched, which requirements remain unknown, and how the buyer can broaden or narrow the request. When strong signals conflict, ask for clarification instead of manufacturing confidence.

Do not compress these signals into one universal similarity score. A global blend hides the distinction between “looks similar,” “has the same model number,” and “satisfies the required specification.” Those statements have different meanings and different failure costs.

The catalog needs a corresponding product layer. For each selected category, create:

• A canonical product entity separate from the seller listings or offers that represent it.
• A category-specific attribute schema, including identity keys and compatibility-critical fields.
• Normalized values linked back to the original listing text.
• Field-level provenance and confidence for inferred or extracted attributes.
• A distinct state for “unknown,” rather than treating missing data as “no.”
• Groups for duplicate or near-duplicate offers so one product does not occupy the entire result page.

That distinction between false and unknown is especially important in an unstandardized marketplace. If a listing does not mention a certification, the system has not proved that the product lacks it. It has proved only that the requirement cannot yet be verified. Exclude it from a confirmed-compliance lane, label the uncertainty, or ask the supplier for the missing field. Do not quietly treat absence as a match.

When a marketplace operates across tens of millions of mostly unstandardized listings, candidate generation must be efficient, but scale does not change this logical order. Vector search can retrieve by appearance, identifier and text indexes can retrieve identity candidates, and structured indexes can enforce normalized constraints. The intent-conditioned re-ranker should combine those candidate sets only after their roles are explicit.

I would require a low-confidence and conflicting-evidence state before approving a launch. “I found visually similar items, but I could not verify the model number” is a useful result. Presenting the same items as exact matches is a product defect, even if the images look excellent.

Launch one category and evaluate the failure modes

The sensible first release is not a catalog-wide image button backed by a general embedding model. Choose one category where the buyer’s decision can be represented clearly and where your team can judge whether a result is exact, compatible, visually analogous, or unacceptable.

A useful launch category has recognizable identity fields, a manageable set of compatibility attributes, sufficient offer coverage after duplicate grouping, and downstream actions you can observe. Avoid beginning with a category where a false compatibility claim could create consequences your data cannot control.

Build the evaluation set from actual buyer uploads collected under your marketplace’s consent, retention, and access policies. Studio product images are helpful catalog assets, but they do not reproduce the screenshots, labels, occlusions, clutter, and mixed text-image evidence the product must interpret.

Label every evaluation query with:

• Image provenance and the selected target region.
• The intended mode: identity, compatibility, appearance, or exploration.
• The decisive evidence, such as a part number, brand and variant, physical feature, or specification.
• Hard constraints and acceptable unknowns.
• An acceptable result set, not just a single presumed answer.
• At least one visually tempting but unacceptable result where such a near miss exists.
• Whether failure is caused by retrieval or by missing and inconsistent catalog data.

The hard negative is important. If your evaluation contains only a correct product and unrelated distractors, a visual model can look capable without learning the distinction the buyer cares about.

Intent mode	Primary offline check	Critical guardrail	Useful online evidence
Exact identity	Entity-resolution accuracy and recall at the evaluated result depth	Wrong product, model, or variant appearing as an exact match	Use of the exact-match lane, reformulation, shortlist, and supplier inquiry
Compatible alternative	Recall of eligible products under known constraints	Hard-constraint violations and unsupported compatibility claims	Constraint edits, comparison, shortlist, and qualified inquiry
Visual analogue	Human-graded ranking quality, such as NDCG at the evaluated result depth	Wrong category or an identity claim inferred from appearance alone	Saves, shortlists, result expansion, and attribute refinement
Category exploration	Category relevance, coverage, and useful result diversity	Duplicate listings crowding out meaningful alternatives	Filter use, query refinement, shortlist, and continued exploration

Do not collapse those rows into one average relevance score. An aggregate can improve while exact-identity queries become less safe, especially if visual-analogue traffic is easier or more common in the evaluation set. Report performance by provenance, intent mode, category, identifier presence, and catalog-data completeness.

Clicks are diagnostic signals, not ground truth. A buyer may click because the result is confusing, because the image is attractive, or because the system returned nothing better. Combine clicks with mode switches, reformulations, comparisons, shortlists, inquiries, and explicit corrections. For controlled experiments, choose the primary outcome that matches the intent lane rather than applying one conversion metric to every visual query.

Review failures using a taxonomy that points to an owner and a fix:

• Target-selection failure: the system searched for the accessory, background object, or package instead of the intended product.
• Routing failure: an identity request was handled as appearance search, or an exploratory request was narrowed to one product.
• Extraction failure: important text, an identifier, or an attribute was missed or misread.
• Entity-resolution failure: extracted evidence was not mapped to the correct canonical item or variant.
• Catalog failure: the relevant offer exists, but the listing lacks normalized attributes or a canonical relationship.
• Constraint failure: a must-have requirement was treated as a soft ranking preference.
• Ranking failure: the right candidates were retrieved but ordered for the wrong intent.
• Offer-grouping failure: duplicate seller listings displaced useful alternatives.

This taxonomy prevents every miss from becoming a request for a larger model. If the identifier was extracted correctly but no listing carries a resolvable identifier, the next investment belongs in catalog normalization or supplier data collection. If the right item was retrieved and then buried, the problem is ranking. If the buyer wanted compatibility but the schema cannot express mounting type, the problem is the product model.

Key takeaways

• Visual search in a B2B marketplace is an intent-resolution problem before it is an image-similarity problem.
• Preserve query provenance because a camera photo, listing screenshot, package, and model plate carry different kinds of evidence.
• Use a query contract to separate exact identity, compatibility, visual analogy, hard constraints, soft preferences, and commercial requirements.
• Route label- and identifier-heavy images through text extraction and entity resolution; visual resemblance must not override product identity.
• Keep exact, structured, textual, and visual retrieval as distinct channels until an intent-conditioned ranking stage combines them.
• Evaluate by query mode and failure type. A single average relevance score will not tell you whether the system is making commercially meaningful mistakes.

Your next useful artifact is not another model benchmark. Review a batch of real visual queries and label three things for each one: where the image came from, how broad the acceptable result set is, and which evidence decides correctness. If knowledgeable reviewers cannot agree on those labels, the product has not yet defined the query well enough for a model to solve it.

Once that query-intent matrix exists, choose one category, build the contract and failure-aware evaluation set, and ship a path that lets buyers correct the system. Better perception will help over time. A product that knows what evidence matters will benefit from every model improvement instead of repeating the same ambiguity with a newer model.

References

• KnowDis AI – KnowDis AI Newsletter- July 2026

---