🐛 [TZF-260067]: Fix 'bytes.seek' error in image search with remote images

[Ravindrayadav04]

Describe the bug

While performing image search using SigLIP embeddings, the API fails with:
'bytes' object has no attribute 'seek'
This occurs both in local and production environments

why this is happening (root cause )

• SigLIP attempts to automatically load images from image_uri
• For S3/CDN URLs, the loader returns raw bytes
• PIL (Image.open) expects a file-like object
• This leads to failure when .seek() is called internally

What did you expect to happen?

The image search API should successfully process images from S3/CDN URLs, generate embeddings using SigLIP, and return relevant search results without errors.

Specifically:

• Images should be properly loaded as valid PIL image objects
• Embeddings should be generated without failure
• The search endpoint should return similar images based on the query

No runtime errors (such as 'bytes' object has no attribute 'seek') should occur during this process.

Sample code link

No response

System info

No response

What browsers are you seeing the problem on?

No response

Relevant log output

Assignment

This issue is free for anyone to take

Before submitting

• I've made research efforts and searched the documentation
• I've searched for existing issues

[github-actions]

👋 Thank you for opening this issue! We will look into it as soon as possible.

[github-actions]

🚀 Branch Created: bugfix/TZF-260067 by @Ravindrayadav04

[Ravindrayadav04]

vectorsearch.md

[Ravindrayadav04]

Vector search

It is also known as the similarity search or nearest neighbor search.
It is a technology that identifies similar items in a dataset by representing data as numerical vectors and comparing them mathematically, allowing systems to understand underlying semantic relationships. Unlike traditional keyword search that relies on exact matches, vector search finds results based on context and meaning, making it ideal for unstructured data such as text, images, and audio.

History of vector search

1960s–1970s:

• Foundations of information retrieval were established.
• Research focused on factor analysis for document classification.
• Early dense and sparse vector representations were developed.
• Key contributions by Gerard Salton (1978) formalized these ideas.

1980s:

• Introduction of Latent Semantic Indexing (LSI).
• Enabled lower-dimensional representations of large document-term matrices.
• Improved search relevance and computational efficiency.

2000s–2019s (Last 20 Years):

• Emergence and growth of vector database management systems (DBMS).
• Significant improvements in scalability and high-performance similarity search.
• Enabled handling of large-scale unstructured data.

2019–Present:

• Integration of advanced machine learning techniques.
• Rise of Transformer-based models like BERT.
• Widespread adoption in AI applications, especially Retrieval-Augmented Generation (RAG).
• Enabled more context-aware, accurate, and intelligent search systems.

keywords and components :

• vector space:
• Vector Embeddings:
• Distance Metrics:
• ANN Search:

Vector Embeddings:

Vector Embeddings: It is a kind of numerical array (vector) which represents the meaning or features of data (e.g., text, image) in a high-dimensional space

Embeddings
A way to convert data (text, image, etc.) into a list of numbers so a computer can understand its meaning.
Example :
input : text --> "red shirt"
Embedding (vector): [0.12, -0.45, 0.88, 0.03, ...]

Types of Embeddings

• Text embeddings → for search/chatbots

• Image embeddings → for image similarity

• Audio embeddings → for voice/music

Distance Metrics :

Distance Metrics (Similarity Indexes) in Vector StoresDistance metrics in vector search are mathematical formulas that measure the similarity or dissimilarity between vector embeddings, directly impacting the accuracy of semantic search, recommendations, and clustering.

The common metrics includes :

Cosine similarity: It is a widely used similarity metric that determines how similar two data points are based on the direction.
It Measures the cosine of the angle between two vectors. Ideal for NLP and semantic search where orientation matters more than magnitude.

Euclidean Distance (L2) : The Euclidean Distance (
norm) is a metric that measures the straight-line distance between two points in a multi-dimensional vector space.
it is use to measure the dismilarity between the embeddings,

|- on the basis of distance we found the similarity like the small distance the similar the objects, and far are dissimilar.

|- it is best used when the magnitude of the vectors is a significant factor in determining similarity.

Dot Product (Inner Product) :

|-It measures both the angle and magnitude of vectors.

|- higher values indicate greater similarity.

|- It provides provide faster and efficient similarity computation

Manhattan distance(L1 distance):

It measures measures how far apart two points are by summing the absolute differences of their coordinates.

ANN Search

Vector indexing

It is a technique used in vector databases to structure high-dimensional vector embeddings, enabling rapid similarity searches by organizing data into efficient structures like trees or graphs.

why we use the vector index --

• purpose : because it replaces the slow brute-force(linear) approach search with Approximate Nearest Neighbor (ANN) search, identifying similar vectors without checking every entry.
• how it works : It maps high-dimensional vectors to special data structures (trees, graphs, or hash tables) that cluster similar data points together.

Common vector index techniques:

• HNSW : for high-speed graphs,
• IVF : for clustered searching,
• LSH : for hashing-based bucketing,
• Flat : for exact, small-scale indexing

IVF : Inverted file

• it is most basic techniques which divides all data in small-small clusters using k-means clustering.
• in which is vector is assign where similar items are kept close together, for making searches faster.
• When you search, the system checks only the closest group instead of the whole data to quickly find the best match.

There are three variants of IVF:

1. IVFFLAT: in this variant each cluster uses a flat structure for stored in a simple list or array without additional subdivision or hierarchical structures. When a query vector is assigned to a cluster, the nearest neighbor is found by conducting a brute-force search, examining each vector in the cluster’s list and calculating its distance to the query vector.this varinat is responsilbe for speed and accuracy.
2. IVFPQ(Inverted File with Product Quantization):It also splits the data into clusters but each vector in a cluster is broken down into smaller vectors, and each part is encoded or compressed into a limited number of bits using product quantization.
3. IVFSQ:(File System with Scalar Quantization):in this variant for every dimension of a vector, we set a predefined value or range. These values or ranges help decide which cluster a vector belongs to. Each component of the vector is then matched against these predefined values to find its place in a cluster. This method of breaking down and quantizing each dimension separately makes the process more straightforward.

HNSW:Hierarchical Navigable Small World Algorithm :

use for storing and fetching the data .
uses the graph structure
it inspired by two techinques --- 
1. probability skip list  
2. Navigable Small World (NSW).

so before understanding the HNSW we have to understand this

1. Skip list : it is advanced data structure which combines two structures

• the quick insertion capability of a linked list and the rapid retrieval characteristic of an array.
  It achieves this through its multi-layer architecture where the data is organized across multiple layers, with each layer containing a subset of the data points.

mechanism and workflow

• Starting from the bottom layer, which contains all data points, each succeeding layer skips some points and thus has fewer data points, ultimately the topmost layer will have the smallest number of data points.
• To search for a data point in a skip list, we start from the highest layer and go from left to right exploring each data point. At any point, if the queried value is greater than the current datapoint, we move back to the previous datapoint in the layer below and resume the search from left to right until we locate the exact point.

2. NSW (Navigable Small World ):

• It is similar to a proximate graph where nodes are linked together based on how similar they are to each other.
• it uses the search for nearest neighnour point .
• it uses the predefined values ..which use as the connecting the multiple nearby nodes.
• when we query , the vectors searches for nearby connecting vector and iterates similar to that ..if no nearby nodes found it stops the search.

Now how HSNW developed ::

• HNSW create the layer like skip list and for the connection multiple data-points it uses the graph-like connection for nodes .the nodes connected to same layer as well as the lower layer. the intensity increases as the we goes to lower layer.

How Does a Search Query Work in HNSW

• Start at the top layer (fast lane) and move forward.
• If a closer option is found below, go down a level.
• Keep repeating this until you reach the bottom layer.
• The bottom layer has all data, so you get the most accurate result.

Note: apart from this In HNSWFLAT, the raw vectors are stored as they are, while in HNSWSQ, the vectors are stored in a quantized form and other methods are same .

vector search Dimensions:

In vector search, dimensions represent the number of floating-point numbers (length) in a vector, which corresponds to the number of semantic features an embedding model has captured about the data.
these are like - 512, 768,1024,1568

Typical Dimension Ranges

• Small Models (128–384 dim): Efficient for quick search, simple semantic tasks.
• Medium Models (512–768 dim): Balanced for most semantic search and RAG pipelines.
• Large Models (1024–1536+ dim): Powerful for highly nuanced data, long documents, or complex image-text relationships.

When to Use Which

• Use 384-512: for building a fast, small-scale chatbot or using low-latency mobile apps.
• Use 768-1024: For RAG systems requiring precise retrieval from large, complex documents.
• Use 1536+: When we using high-end models like text-embedding-3-large then for maximum semantic understanding we use these kind of things.