Code10x

Ek distributed search aur analytics engine hai, Apache Lucene par bana — full-text search, log analytics aur real-time data exploration ke liye use hota hai.

Shay Banon ne 2010 mein launch kiya tha — originally apni wife ke liye ek recipe search engine banane ki koshish se shuru hua tha.

Dono Apache Lucene par bane hain, isliye core search/indexing capabilities largely similar hain.

Elasticsearch "distributed-first" philosophy ke saath design kiya gaya tha — JSON REST APIs aur horizontal scaling day 1 se core mein the.

ELK Stack (Elasticsearch, Logstash, Kibana) — log aggregation aur observability ke liye industry standard ban gaya hai.

RESTful HTTP API — har operation ek HTTP method (GET, POST, PUT, DELETE) aur ek URL path se hota hai.

JSON — ye Elasticsearch ko kisi bhi programming language se easily integrate karne layak banata hai.

Cluster (ek ya more nodes ka group) > Node (ek Elasticsearch server instance) > Index (documents ka collection) > Shard (index ke data ka ek hissa).

Client HTTP request bhejta hai -> koi bhi node request receive karta hai (coordinating node) -> relevant shards ko forward karta hai -> results aggregate hote hain -> response client ko jaata hai.

TAR/ZIP archive download karke, package manager (apt/yum) se, ya Docker image se — bundled JDK ke saath aata hai.

8.x se security (TLS, passwords) default enabled ke saath ship hoti hai — pehle 7.x mein security default disabled thi.

Decide karta hai ki koi node kis cluster mein join karega — same cluster.name waale saare nodes hi ek cluster mein join kar sakte hain.

Decide karti hain ki index data aur log files kahan store hongi — production mein inhe dedicated disk par point karna recommend kiya jaata hai.

Official "elasticsearch" Docker image se ek hi command mein run kiya ja sakta hai — local development ke liye popular tareeka hai.

Local/dev environments mein — ye Elasticsearch ko batati hai ki kisi doosre node ke saath cluster form karne ki koshish mat karo.

Master (cluster state manage karta hai), Data (actual data store karta hai), Ingest (documents transform karta hai), aur Coordinating (requests route karta hai).

Large production clusters mein cluster ki stability ke liye — ye isolate karta hai ki heavy data operations cluster management ko affect na karein.

"Split-brain" scenarios ko avoid karne ke liye — majority-based (quorum) decisions lene mein help karta hai.

Documents ka ek collection jo similar characteristics share karte hain — relational database mein "table" ke equivalent hai.

Lowercase honi chahiye, special characters (jaise \ / * ? " < > | space) allowed nahi hain.

Full-text search ke liye — value analyzed (tokenized, lowercase) hoti hai indexing ke waqt.

Exact-match, sorting aur aggregations ke liye — raw value as-is store hoti hai, bina analysis ke.

Elasticsearch khud decide karta hai ki ek naye field ka type kya hoga, JSON value ke based par (number, text, boolean waghera).

Kyunki agar ek field ka format inconsistent hai (jaise date kabhi string, kabhi number), to wrong type infer ho sakta hai aur baad mein queries fail ho sakti hain.

Ek pattern (jaise "logs-*") define karte hain — jab bhi naya index us pattern se match karta hai, Elasticsearch automatically specified mappings/settings apply kar deta hai.

Multiple Component Templates (reusable mapping/settings blocks) ko combine kar sakte hain — DRY principle ko index configuration mein apply karta hai.

"fields" parameter — jaise "title" (text) aur "title.keyword" (keyword) dono ek hi source se.

POST /_doc se Elasticsearch automatically unique ID generate karta hai, ya PUT /_doc/<id> se explicit ID specify ki ja sakti hai.

PUT idempotent hai — same ID se dobara PUT karne se existing document overwrite ho jaata hai. POST (bina ID) har baar naya document banata hai.

Kyunki GET by ID directly document retrieve karta hai, koi relevance scoring ya query parsing nahi hoti.

Ek document ke sirf specific fields ko update karta hai — Elasticsearch internally poore document ko fetch, changes apply, aur dobara index karta hai.

Kyunki Lucene documents immutable hain — har update purane document ko delete-mark karke ek naya version banata hai.

Multiple index/update/delete operations ek hi HTTP request mein bheje ja sakte hain — network overhead kaafi kam karta hai.

NDJSON (Newline-Delimited JSON) — har action line ke baad newline hona zaroori hai, last line ke baad bhi.

Ek existing index ki mapping/settings change karna mushkil hai — reindex se old index se data ko naye index mein copy kiya ja sakta hai naye mapping ke saath.

Ek alias use karke jo actual index ki taraf point karta hai — reindex complete hone ke baad alias ko naye index par atomically switch kar diya jaata hai.

"text" fields par — search term ko usi analyzer se guzara jaata hai jo indexing ke waqt use hua tha, user-typed searches ke liye ideal hai.

Exact value match ke liye, bina analysis ke — "keyword" fields, boolean fields ya numbers par use hoti hai.

Kyunki indexed value lowercase tokens mein hoti hai, term query exact original string dhoondti hai jo match nahi karegi.

must (AND with scoring), should (OR — matching par score badhta hai), must_not (NOT), aur filter (AND, no scoring effect).

Filter clauses mein koi relevance scoring nahi hoti isliye results cache ho sakte hain.

BM25 relevance scoring algorithm ke hisaab se — jitna zyaada relevant document, utna high score.

Keyword search binary hai — ya to exact match hai ya nahi hai, koi partial match ya relevance scoring variation nahi hoti.

Kyunki invalid syntax par ye exception throw karti hai — user ki typos se application crash ho sakta hai.

Kyunki invalid syntax par exception throw nahi karti, balki galat parts ko ignore kar ke valid parts ko hi process karti hai.

Ek field ke unique values ke hisaab se documents ko "buckets" mein group karti hai, har bucket ka count deti hai.

Time-based buckets (jaise per-day, per-hour) banati hai — log analytics dashboards ka foundation hai.

avg, sum, min, max, value_count — ek numeric field par simple mathematical calculations karte hain.

Ek field ke unique values ki approximate count, HyperLogLog++ algorithm use karke — exact count large datasets par memory-intensive hota hai.

Ek bucket aggregation (jaise terms) ke andar ek metric aggregation (jaise avg) — har category ke andar uski average price milti hai.

Default mein aggregation sirf us query se match karne waale documents par hi calculate hoti hai.

Ek filter apply karta hai jo sirf hits ko affect kare, aggregations ko nahi — e-commerce faceted search mein common pattern hai.

Doosri aggregations ke output par kaam karti hain, na ki documents par directly — jaise "derivative" ya "moving_avg".

Parent aggregation ke naam se hi child aggregation ka path banata hai (jaise "sales_per_month>revenue") — pipeline aggregations mein use hota hai.

Ek ya more nodes ka group jo same "cluster.name" share karte hain — data aur workload ko distribute karte hain.

Cluster-wide decisions leta hai — jaise naya index banana, shards ko move karna.

Baaki master-eligible nodes automatically ek naya master elect kar lete hain — ye process usually seconds mein hota hai.

Index ke data ka ek hissa jo different nodes par distribute ho sakta hai, parallelism aur scale ke liye.

Ek primary shard ki copy, alag node par — node down hone par bhi availability maintain rehti hai.

Primary shard count index create hone ke baad change nahi ki ja sakti, lekin replica count kabhi bhi dynamically change ho sakti hai.

GREEN (sab shards active), YELLOW (primary active, replicas missing), RED (primary shards bhi unavailable).

Network partition ki wajah se do alag groups of nodes dono khud ko "master" samajhne lagte hain — data inconsistency ka cause ban sakta hai.

Kyunki majority (quorum) calculate karna easy hai — even numbers tie situations create kar sakte hain.

Ek web-based visualization aur exploration tool jo Elasticsearch ke data par kaam karta hai — charts, dashboards, maps banane ke liye.

Batata hai ki kaunse indices (jaise "logs-*") par Kibana ko query chalani hai, visualizations banane se pehle.

Directly Elasticsearch ki raw REST API ko test karne deta hai, JSON autocomplete ke saath.

Input (data kahan se aa raha hai), Filter (data transform/parse karna), aur Output (processed data kahan jaayega).

Unstructured log lines ko structured fields mein parse karta hai predefined patterns (jaise %{IP}, %{TIMESTAMP_ISO8601}) ke zariye.

Lightweight data shippers ka family — jaise Filebeat (logs), Metricbeat (metrics), Packetbeat (network traffic) — Go mein likhe gaye, minimal resources use karte hain.

Simple data shipping ke liye Beats, complex transformation logic ke liye Logstash — common pattern hai Beats -> Logstash -> Elasticsearch.

Elasticsearch (storage/search), Logstash (data processing), Kibana (visualization) — ab "Elastic Stack" kehlaata hai kyunki Beats bhi shamil hai.

Centralized log aggregation, Application Performance Monitoring (APM), aur Security Information & Event Management (SIEM).

Jab dynamic mapping ke saath thousands of unique field names automatically create ho jaate hain — cluster performance aur memory ko severely impact kar sakta hai.

Ek index mein maximum allowed fields ki count restrict karta hai (default 1000) — mapping explosion se protect karta hai.

Kyunki document index karne ke turant baad wo search mein nahi dikhta — naye documents sirf har "refresh" (default 1 second) ke baad hi searchable hote hain.

"-1" (disabled) karne se indexing speed significantly badh sakti hai, kyunki har refresh ek naya Lucene segment banata hai jisme CPU cost lagti hai.

Ek external "repository" (jaise S3 bucket ya shared filesystem) mein — incremental hote hain, sirf changed data transfer hota hai.

Pehle ye separate "X-Pack" paid add-on tha, ab built-in aur default enabled hai — authentication, authorization, TLS shamil hai.

Kyunki inhe expire kiya ja sakta hai aur granularly scope kiya ja sakta hai, username/password ke bajay.

Cluster health, node performance, indexing rates aur search latency ko Kibana ke andar hi visualize karta hai.

Kyunki disk full hone par Elasticsearch indices ko read-only bhi kar sakta hai — early warning ke liye 85%/90%/95% thresholds set kiye jaate hain.