ZVEC + Proxima: Alibaba's In-Process Vector Database for Domain Intelligence
TL;DR: ZVEC and Proxima are Alibaba's open-source vector search technologies. Unlike client-server vector databases (Pinecone, Qdrant), ZVEC runs in-process — no server, no network latency. Sub-millisecond queries on billions of vectors. Free. Perfect for dom.to's 103M domains and AIM.in's intent-based matching.
What is ZVEC?
ZVEC is a lightweight, in-process vector database built on Proxima — Alibaba's battle-tested vector search engine powering Taobao search, Alipay face payments, Youku video search, and Alimama advertising.
Key Features
| Feature | Description |
|---|---|
| In-Process | No server, daemon, or infrastructure. Import and use. |
| Blazing Fast | Sub-millisecond searches on billions of vectors |
| Dense + Sparse | Both embedding vectors AND keyword/BM25 sparse vectors |
| Hybrid Search | Combine semantic similarity with structured filters |
| Quantization | INT8 quantization reduces memory by 4x with <1% recall loss |
| Cross-Platform | Linux x64/ARM64, macOS ARM64. Python 3.10-3.12, Node.js |
Installation
# Python pip install zvec # Node.js npm install @zvec/zvec
That's it. No Docker, no config files, no cloud setup.
What is Proxima?
Proxima is the underlying engine — a production-grade vector search system that has handledtrillions of queries at Alibaba.
Production Deployments
| System | Scale | Use Case |
|---|---|---|
| Taobao Search | Billions of products | Product similarity, recommendations |
| Alipay Face Pay | Hundreds of millions of faces | Real-time face matching |
| Youku | Billions of video frames | Visual search, deduplication |
| Alimama Ads | Trillions of ad-query pairs | Real-time ad ranking |
Benchmark Performance
Using VectorDBBench on standardized Cohere datasets:
| Dataset | QPS | Recall@10 | Memory (INT8) |
|---|---|---|---|
| Cohere 10M (10M × 768d) | ~4,000+ | 95%+ | ~4GB |
| Cohere 1M (1M × 768d) | ~10,000+ | 97%+ | ~400MB |
Application 1: dom.to Domain Intelligence
Current State
dom.to has:
- 103M+ domains in PostgreSQL
- 3M+ .in domains with detailed intelligence
- 1.9M screenshots for visual analysis
- 38M WHOIS records
What ZVEC Enables
1. Semantic Domain Search
# Before: Exact match or LIKE queries
SELECT * FROM domains WHERE name LIKE '%startup%';
# After: Semantic similarity
results = zvec_index.query(
vector=embed("fintech startup"),
topk=100,
filter={"tld": "in", "available": True}
)
# Returns: paymentgateway.in, lending.in, neobank.co.in, ...2. Multi-Vector Domain Profiles
# Each domain has multiple embeddings
schema = zvec.CollectionSchema(
name="domains",
vectors=[
zvec.VectorSchema("name_embed", FP32, 768), # Domain name
zvec.VectorSchema("content_embed", FP32, 768), # Page content
zvec.VectorSchema("visual_embed", FP32, 512), # Screenshot CLIP
zvec.VectorSchema("category_sparse", SPARSE), # Industry keywords
]
)3. Hybrid Search with Filters
results = index.query(
vectors={
"name_embed": embed("ecommerce platform"),
"category_sparse": sparse_encode(["marketplace", "B2B"])
},
filter={
"tld": {"$in": ["in", "co.in", "com"]},
"registrar": {"$ne": "GoDaddy"},
"registered_date": {"$gt": "2024-01-01"}
},
topk=100
)Application 2: AIM.in Intent-Based Matching
The Vision
Traditional B2B marketplaces show listings. Buyers browse, filter, compare.
AIM.in vision: Intent-based matching. Buyer describes what they need → AI matches to best suppliers with explanations.
# Buyer query
query = "Need 500mm RCC pipes, 100 units, delivery to Madhapur"
# Vector search with geo-filter
results = supplier_index.query(
vector=embed(query),
filter={
"delivery_zones": {"$contains": "Telangana"},
"product_categories": {"$contains": "RCC pipes"},
"min_order_qty": {"$lte": 100}
},
topk=10
)
# Result with explanation:
# ✅ Vijaya RCC (94% match)
# - Product: RCC pipes 500mm available ✓
# - Location: 12km from Madhapur ✓
# - Capacity: Can deliver 100 units in 3 days ✓Why ZVEC Over Alternatives?
vs. Pinecone/Qdrant/Weaviate
| Factor | ZVEC | Cloud Vector DBs |
|---|---|---|
| Latency | <1ms (in-process) | 10-100ms (network) |
| Cost | Free (self-hosted) | $70-700/mo at scale |
| Privacy | Data stays local | Data leaves your infra |
| Scale | Billions (single node) | Millions (free tier) |
vs. pgvector
| Factor | ZVEC | pgvector |
|---|---|---|
| Performance | 10-100x faster at scale | Degrades with data size |
| Hybrid Search | Native dense + sparse | Requires workarounds |
| Quantization | INT8 built-in | Limited |
Implementation Roadmap
| Phase | Scope | Timeline |
|---|---|---|
| Phase 1 | Index 3M .in domains with name embeddings | Week 1 |
| Phase 2 | Add screenshot embeddings (1.9M images) | Week 2-3 |
| Phase 3 | Hybrid search with WHOIS filters | Week 4 |
| Phase 4 | Scale to full 103M corpus | Week 5-6 |
Memory Estimate
103M domains × 768 dims × 4 bytes = ~316GB (FP32) With INT8 quantization: ~79GB With disk-based index: Fits on standard VPS
Conclusion
ZVEC + Proxima gives us:
- dom.to: Semantic domain search across 103M domains at sub-millisecond latency
- AIM.in: Intent-based supplier matching with explainable AI
- Cost: Zero (vs. $200-500/mo for cloud vector DBs at our scale)
- Control: Data stays on our infrastructure
Recommended next step: Pilot on dom.to with 3M .in domains. 2-week timeline. Measure latency improvement over current PostgreSQL approach.
References
Research by OpenGarage • Published 2026-02-21 • Get in touch to discuss implementation