Amazon OpenSearch Service

By the end of this lesson you will be able to

• Explain why a search engine finds words, parts of words and typos in milliseconds when a database cannot, using the inverted index, analysers and relevance scoring.
• Describe Amazon OpenSearch Service: an open-source fork of Elasticsearch, run as a managed domain (data nodes, shards, replicas, Multi-AZ) or as OpenSearch Serverless (collections and OCUs), with OpenSearch Dashboards on top.
• Recognise the exam keywords that point to OpenSearch (full-text search, log analytics, real-time application monitoring, Kibana) and use it as a complement to the source-of-truth database, never instead of it.
• Secure a domain with IAM and access policies, VPC access, fine-grained access control, Cognito or SAML for Dashboards, KMS at rest and TLS in transit.
• Build the three ingestion patterns from DynamoDB Streams, CloudWatch Logs and Kinesis, choose between Lambda (real time) and Firehose (near real time), and say what happens when each one fails.
• Place the 2023–26 additions correctly: OpenSearch Ingestion, zero-ETL with DynamoDB and S3, and vector search for generative AI.

#Why a search engine at all

#What databases are bad at

Every database you have met so far finds data fast only by the path it was built for:

• DynamoDB answers by partition key (and sort key), or through a secondary index you created in advance (lesson 15.5). Ask it for “every product whose description mentions katan” and the only tool left is a Scan that reads the whole table.
• RDS and Aurora have B-tree indexes on columns. They work for WHERE id = 42 or WHERE title LIKE 'metro%', but not for WHERE body LIKE '%metro%': a pattern that starts with a wildcard cannot use the index, so the database reads every row. And LIKE only compares characters: it does not know that “floods” and “flood” are the same word, that “kaatan” is a typo for “katan”, or which of a thousand matches is the best one.

People type into search boxes the way they talk: partial words, wrong spellings, words in any order, and they expect the best result first. That is a different problem from “fetch the row with this key”, and it needs a different data structure.

#The inverted index

A search engine turns the problem around. When a document is stored, its text goes through an analyser:

1. Tokenise: split the text into words.
2. Normalise: lowercase, remove accents, drop very common stop words (“the”, “in”) if configured.
3. Stem: reduce words to a root, so “floods”, “flooded” and “flooding” all become “flood”.

Each resulting term is written into an inverted index: a sorted list of every term, each pointing to the documents (and positions) that contain it. A search analyses the query the same way and then just looks up its terms, which takes the same few milliseconds whether there are ten documents or ten billion.

On top of that lookup the engine adds what users expect:

• Partial matches and autocomplete: walk the sorted term list from a prefix (“metr” → “metro”), or index word fragments (edge n-grams) so suggestions appear while typing.
• Fuzzy matching: accept terms one or two edits away (“criket” → “cricket”).
• Relevance ranking: score each match with BM25 (words that are rare across all documents and appear in short fields count more) and return the best first.
• Faceted filtering and aggregations: “Saree (124) · Panjabi (37) · under 5,000 taka (61)” counts next to the results, and the funnel of narrowing them down by clicking. The notes call this funnel filtering.
• Highlighting, synonyms, geo search, and analysers for many languages, including Bengali.

#From Elasticsearch to OpenSearch

Elasticsearch (with Logstash for collecting logs and Kibana for dashboards, together the ELK stack) was for a decade the most popular open-source search engine, and AWS ran it as Amazon Elasticsearch Service. In January 2021 Elastic moved new versions to licences that are not open source. AWS and others then forked the last open-source versions (Elasticsearch 7.10.2 and Kibana 7.10.2) into OpenSearch and OpenSearch Dashboards, under the Apache 2.0 licence. In September 2021 the service was renamed Amazon OpenSearch Service. Since 2024 the project belongs to the OpenSearch Software Foundation under the Linux Foundation.

What that means for you today:

• OpenSearch Service runs OpenSearch (1.x, 2.x and 3.x versions) and still supports legacy Elasticsearch domains up to 7.10, the last open-source release. Many old versions left standard support in November 2025 and now cost an extended support fee per instance hour, so upgrade old domains.
• OpenSearch Dashboards is the successor of Kibana. On a legacy Elasticsearch domain the dashboard tool is still called Kibana, which is why exam questions still say “Kibana dashboard”: the answer is OpenSearch Service.
• APIs and clients of Elasticsearch 7.10 mostly work unchanged, so migrating an old ELK setup is usually simple.

#How a domain stores and serves data

#Documents, indexes, shards and nodes

• A document is one JSON object: an article, a product, a log line. Each has an ID.
• An index is a collection of similar documents (news, products, payments-2026.10.30), with a mapping that says what type each field is (text, keyword, number, date, geo point, vector). OpenSearch guesses the mapping from the first document if you do not set one, and a field keeps its type: send a number first and a string later, and the later document is rejected.
• An index is split into primary shards, so that its data and work can spread across machines. Each primary can have replica shards: full copies on other nodes. Writes go to the primary, then to its replicas; searches can be answered by any copy, so replicas add both durability and read capacity. The number of primary shards is fixed when the index is created (changing it needs a split, shrink or reindex); the number of replicas can change at any time.
• A domain is AWS’s name for an OpenSearch cluster: a set of data nodes (EC2 instances that hold shards, up to 1,002 per domain) and optional dedicated cluster manager nodes (formerly “master” nodes; three are recommended) that keep track of the cluster state and decide where shards live, without storing data themselves.
• Writes become searchable after a refresh, every 1 second by default. OpenSearch is near real-time, not instant.

The cluster reports its health as a colour: green (every primary and replica is assigned), yellow (all primaries are there but some replicas are missing: no data is lost, but there is no spare copy), red (at least one primary has no working copy: part of the data is unavailable).

Multi-AZ. OpenSearch Service can spread nodes over two or three Availability Zones and puts a shard’s replica in a different AZ from its primary, so an AZ outage loses nothing. Multi-AZ with Standby goes further: nodes in three AZs, one of them a standby that takes over in seconds, for a 99.99% availability SLA. A domain lives in one Region; for another Region, use cross-cluster replication or snapshots.

Backups. The service takes automated snapshots every hour and keeps them for 14 days, free, in storage you do not see. For restores in another domain or Region, take manual snapshots into your own S3 bucket.

#Querying: Query DSL, SQL and PPL

The native way to search is the Query DSL: a JSON request with match (analysed full text), term (exact value), prefix, fuzzy, range, bool to combine them, and aggregations for counts, averages and histograms (the basis of every dashboard chart).

OpenSearch Service also ships a SQL plugin in every domain: you can send SELECT ... FROM payments WHERE ... to the _plugins/_sql endpoint, use PPL (Piped Processing Language, source=payments | where status=500 | stats count() by host) for logs, and connect BI tools through the JDBC driver. It is handy, but OpenSearch is not a relational database: no transactions, limited joins.

#Storage tiers for logs that grow every day

Log analytics domains grow forever unless you manage them. OpenSearch Service has three tiers:

An Index State Management (ISM) policy moves indexes between tiers as they age and finally deletes them. Without one, the disk fills up, and when free space runs out OpenSearch Service blocks writes to protect the cluster.

#Two ways to run it: domains and Serverless

The slides call them the two modes: a managed cluster and a serverless one.

Exam rule: “no cluster management, unpredictable or intermittent search or log workloads” → OpenSearch Serverless. “Need a specific instance type, UltraWarm, custom plugins or full control” → a managed domain.

#Security

OpenSearch holds exactly the data attackers like (logs full of user IDs, payment events, personal data), and in 2018–2020 thousands of self-hosted Elasticsearch clusters leaked because they were open to the internet with no password. The managed service gives you layers:

• Network. Choose VPC access (the domain gets network interfaces in your private subnets, guarded by security groups) or a public endpoint guarded by policies. The choice is made at creation and cannot be switched later. Serverless uses network policies (VPC endpoint or public).
• Who may call the API: IAM. A domain access policy (resource-based, like an S3 bucket policy) and identity-based IAM policies decide which principals can call which HTTP methods on which indexes; requests are signed with SigV4. An IP-based policy is possible for public domains, but weak.
• What they may see inside: fine-grained access control (FGAC). Roles inside OpenSearch grant access per index, per document (only rows matching a query) and per field (hide or mask fields such as phone numbers). FGAC requires HTTPS, encryption at rest and node-to-node encryption.
• Who may log in to Dashboards. Amazon Cognito user pools (and identity pools for AWS credentials), SAML federation with your corporate IdP, or the internal user database with HTTP basic authentication.
• Encryption. At rest with KMS (data, logs, snapshots, swap); in transit with TLS: HTTPS required for clients, and node-to-node encryption inside the cluster. Serverless always encrypts, with an AWS-owned or your own KMS key (encryption policy).
• Audit. Configuration API calls go to CloudTrail; FGAC audit logs record who searched what.

#OpenSearch Dashboards and log analytics

OpenSearch Dashboards is the web UI included with every domain and collection: Discover to search and filter raw events, visualisations and dashboards (charts, maps, tables), alerting (monitors that notify through SNS and others when, for example, error count per minute crosses 50), anomaly detection, observability (logs, traces and metrics of instrumented applications together), Security Analytics (SIEM-style detection rules) and Query Workbench for SQL and PPL.

This is why the notes say OpenSearch is used for application logs, system tracking, VPC Flow Logs and real-time monitoring: the data arrives within seconds, any field can be searched, and the same tool draws the dashboard and raises the alert. CloudWatch Logs Insights (lesson 19.4) covers many simple cases without an extra service; OpenSearch wins when you need full-text search across many sources, long retention at lower cost, complex dashboards or a self-service tool for many teams.

#Getting data in: the three patterns

OpenSearch is almost always a copy. The slides say it plainly: it is common to use OpenSearch as a complement to another database. Data is written to the system of record (DynamoDB, Aurora, S3, CloudWatch Logs, a Kinesis stream) and then flows into OpenSearch. Native ingestion paths are Amazon Data Firehose, AWS IoT Core (a rule action that sends device messages straight to a domain), CloudWatch Logs subscriptions and, since 2023, OpenSearch Ingestion. The exam’s three classic patterns follow.

#Pattern 1: DynamoDB → DynamoDB Streams → Lambda → OpenSearch

DynamoDB is brilliant at “give me item P-17” and useless at “find red katan sarees under 8,000 taka”. So:

1. The application writes to the DynamoDB table (create, update, delete: CRUD) as usual.
2. DynamoDB Streams records every item-level change, in order per item, for 24 hours (lesson 15.5).
3. A Lambda function reads the stream through an event source mapping and indexes, updates or deletes the matching document in OpenSearch, using the item’s key as the document ID so that retries overwrite instead of duplicating.
4. The application uses two APIs: an API to search items (OpenSearch returns matching IDs, ranked) and an API to retrieve items (DynamoDB returns the current item by key).

Fetching the final item from DynamoDB is deliberate. The index lags by about a second and may miss a change if the sync fails; price, stock and permissions must come from the source of truth.

#Pattern 2: CloudWatch Logs → subscription filter → Lambda or Firehose

Every AWS service and your own applications can write logs to CloudWatch Logs (lesson 19.2). A subscription filter on a log group forwards matching events (all of them, or only lines matching a pattern such as ERROR) to a destination, compressed with gzip. Today a log group can have up to five subscription filters (older material says two). Two destinations lead to OpenSearch:

#Pattern 3: Kinesis Data Streams → Firehose or Lambda → OpenSearch

When events already flow through Kinesis Data Streams (lesson 13.5), there are again two routes:

• KDS → Firehose → OpenSearch (near real time). Firehose reads the stream as a consumer, optionally runs a Lambda function for data transformation (mask fields, convert formats, add data), buffers and bulk-loads. Records the function marks as failed go to processing-failed/ in S3; documents OpenSearch rejects go to AmazonOpenSearchService_failed/ (lesson 13.6).
• KDS → Lambda → OpenSearch (real time). A Lambda event source mapping reads each shard and your code indexes the batch within about a second. You control everything, and you also own retries, duplicates and poison records (set a retry limit, bisect on error and an on-failure destination).

Because Kinesis keeps records for the retention period, either route can replay after a fix, and other consumers (fraud detection, an S3 archive) read the same stream independently.

#Other ways in

• OpenSearch Ingestion (2023): a serverless, managed Data Prepper pipeline (Data Prepper is OpenSearch’s open-source data collector, the modern Logstash). Sources include HTTP and trace collectors, S3 (with SQS notifications), DynamoDB, DocumentDB, Kinesis Data Streams and Kafka/MSK; it filters, enriches and routes data into domains or collections. Billed per Ingestion OCU-hour; you can stop a pipeline to save money.
• Zero-ETL with S3, CloudWatch Logs and Security Lake (direct query, 2024): query logs where they already are, without loading them into the domain, and accelerate only the parts you query often. Good for rarely searched archives.
• AWS IoT Core rule action, Logstash or Fluent Bit agents, or your own code with the _bulk API.

#Vector search for generative AI

Besides words, OpenSearch can index vectors: lists of numbers that an embedding model produces to capture the meaning of a text or image. A k-NN (k-nearest neighbours) query returns the documents whose vectors are closest to the query’s, so “cheap flights to Cox’s Bazar” can find an article titled “Budget trips to the sea beach” that shares no words with it. This is semantic search, and it is the retrieval step of RAG (retrieval-augmented generation): find the relevant passages, then give them to an LLM with the question.

Domains have the k-NN plugin; Serverless has a dedicated vector search collection type (the search and time series types cannot use k-NN indexes). Amazon Bedrock Knowledge Bases can create and use an OpenSearch Serverless vector collection for you (lesson 18.2). Exam hint: “vector database / store embeddings for a generative AI application, fully managed” → OpenSearch Serverless (vector search) is a standard answer.

#What goes wrong

#OpenSearch in system design

• It is a secondary store. Make DynamoDB, Aurora or S3 the source of truth and keep OpenSearch rebuildable from it. If the index is lost or corrupted, re-index; never take orders or money from it.
• Search box, log analytics, observability, security analytics, vector search: those are its jobs. Not transactions (Aurora), not key lookups at any scale (DynamoDB), not SQL reports over years of data (Redshift), not occasional SQL over files in S3 (Athena). It also connects to QuickSight for business dashboards (lesson 17.5).
• Pick the ingestion path by latency and code: Firehose (near real time, no code), Lambda (real time, your code), OpenSearch Ingestion / zero-ETL (managed, configured not coded).
• Plan for growth: shard sizes of roughly 10–50 GB, at least one replica, two or three AZs, dedicated cluster manager nodes for production, tiers for logs.

#Try it yourself

# 1. Create a small managed domain (2 data nodes in 2 AZs, encrypted, HTTPS only, FGAC on).
aws opensearch create-domain --domain-name news-example \
  --engine-version OpenSearch_2.19 \
  --cluster-config InstanceType=t3.small.search,InstanceCount=2,ZoneAwarenessEnabled=true,ZoneAwarenessConfig={AvailabilityZoneCount=2} \
  --ebs-options EBSEnabled=true,VolumeType=gp3,VolumeSize=10 \
  --encryption-at-rest-options Enabled=true \
  --node-to-node-encryption-options Enabled=true \
  --domain-endpoint-options EnforceHTTPS=true \
  --advanced-security-options 'Enabled=true,InternalUserDatabaseEnabled=true,MasterUserOptions={MasterUserName=admin,MasterUserPassword=Example-Passw0rd!}' \
  --access-policies '{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Principal":{"AWS":"arn:aws:iam::111122223333:root"},"Action":"es:ESHttp*","Resource":"arn:aws:es:ap-southeast-1:111122223333:domain/news-example/*"}]}'

# Wait until Processing is false and an endpoint appears (10–20 minutes).
aws opensearch describe-domain --domain-name news-example \
  --query 'DomainStatus.{Processing:Processing,Endpoint:Endpoint}'

{ "Processing": false, "Endpoint": "search-news-example-abc123.ap-southeast-1.es.amazonaws.com" }

E=https://search-news-example-abc123.ap-southeast-1.es.amazonaws.com
A='admin:Example-Passw0rd!'

# 2. Create an index with 2 primaries and 1 replica, and an explicit mapping.
curl -s -u "$A" -X PUT "$E/news" -H 'Content-Type: application/json' -d '{
  "settings": { "number_of_shards": 2, "number_of_replicas": 1 },
  "mappings": { "properties": {
    "title": { "type": "text" }, "district": { "type": "keyword" },
    "published": { "type": "date" } } } }'

# 3. Index two documents (bulk) and search with full text, prefix and fuzzy queries.
curl -s -u "$A" -X POST "$E/_bulk?refresh=true" -H 'Content-Type: application/x-ndjson' --data-binary $'
{"index":{"_index":"news","_id":"1001"}}
{"title":"Floods hit Sylhet tea gardens","district":"Sylhet","published":"2026-10-28"}
{"index":{"_index":"news","_id":"1002"}}
{"title":"Dhaka metro rail adds new trains","district":"Dhaka","published":"2026-10-29"}
'
curl -s -u "$A" "$E/news/_search?filter_path=hits.hits._id,hits.hits._score" \
  -H 'Content-Type: application/json' -d '{"query":{"match":{"title":"flood sylhet"}}}'

{"hits":{"hits":[{"_id":"1001","_score":1.3862942}]}}

curl -s -u "$A" "$E/news/_search?filter_path=hits.hits._id" -H 'Content-Type: application/json' \
  -d '{"query":{"prefix":{"title":"metr"}}}'                       # autocomplete-style → 1002
curl -s -u "$A" "$E/news/_search?filter_path=hits.hits._id" -H 'Content-Type: application/json' \
  -d '{"query":{"fuzzy":{"title":{"value":"gardns"}}}}'             # one typo → 1001

# 4. The same data through the SQL plugin, and the cluster's health and shard placement.
curl -s -u "$A" -X POST "$E/_plugins/_sql?format=txt" -H 'Content-Type: application/json' \
  -d '{"query":"SELECT district, COUNT(*) FROM news GROUP BY district"}'
curl -s -u "$A" "$E/_cluster/health?filter_path=status,number_of_nodes,unassigned_shards"
curl -s -u "$A" "$E/_cat/shards/news?v"

{"status":"green","number_of_nodes":2,"unassigned_shards":0}
index shard prirep state   docs node
news  0     p      STARTED    1 9c1f…
news  0     r      STARTED    1 4a7e…
news  1     p      STARTED    1 4a7e…
news  1     r      STARTED    1 9c1f…

# 5. Stream a CloudWatch Logs group into OpenSearch through Firehose (near real time).
#    (Firehose stream "logs-to-os-example" with the domain as destination already exists.)
aws logs put-subscription-filter --log-group-name /payments/api \
  --filter-name to-opensearch --filter-pattern 'ERROR' \
  --destination-arn arn:aws:firehose:ap-southeast-1:111122223333:deliverystream/logs-to-os-example \
  --role-arn arn:aws:iam::111122223333:role/CWLtoFirehoseRole
aws logs describe-subscription-filters --log-group-name /payments/api

# 6. OpenSearch Serverless: a search collection needs an encryption policy first,
#    then a network policy and a data access policy.
aws opensearchserverless create-security-policy --name products-enc --type encryption \
  --policy '{"Rules":[{"ResourceType":"collection","Resource":["collection/products-example"]}],"AWSOwnedKey":true}'
aws opensearchserverless create-security-policy --name products-net --type network \
  --policy '[{"Rules":[{"ResourceType":"collection","Resource":["collection/products-example"]}],"AllowFromPublic":true}]'
aws opensearchserverless create-collection --name products-example --type SEARCH
aws opensearchserverless batch-get-collection --names products-example \
  --query 'collectionDetails[0].{status:status,endpoint:collectionEndpoint}'

# 7. Clean up.
aws logs delete-subscription-filter --log-group-name /payments/api --filter-name to-opensearch
aws opensearchserverless delete-collection --id <collection-id>
aws opensearch delete-domain --domain-name news-example

Also try in the console: open the domain’s OpenSearch Dashboards URL, create an index pattern for news*, and use Discover to filter by district; then open Query Workbench and run the same SQL as above.

#Summary

Key takeaways

• A search engine stores text in an inverted index (term → documents) after analysis (tokenise, lowercase, stem), so it finds words, partial matches (prefix, autocomplete), typos (fuzzy) and ranks by relevance (BM25) in milliseconds. LIKE '%x%' must scan every row and understands none of this.
• Amazon OpenSearch Service = the successor of Amazon Elasticsearch Service (renamed 2021), running OpenSearch (a fork of Elasticsearch 7.10) and legacy Elasticsearch up to 7.10. OpenSearch Dashboards replaces Kibana. Exam keywords: full-text search, log analytics, real-time application monitoring, Kibana, ELK.
• In DynamoDB you query by key or index; with OpenSearch you can search any field, even partially. It is a complement to the source-of-truth database, never the source of truth.
• A domain = data nodes holding primary and replica shards, optional dedicated cluster manager nodes, Multi-AZ (2–3 AZs, Multi-AZ with Standby for 99.99%), hourly automated snapshots. Green / yellow / red health. Near real-time (refresh about 1 s).
• Managed domain (you size nodes, full control, UltraWarm and cold tiers, ISM) vs OpenSearch Serverless (collections: search, time series, vector; OCUs scale indexing and search separately).
• SQL works through the built-in SQL plugin (plus PPL), but the native language is the Query DSL.
• Security: VPC or public access (fixed at creation), IAM and domain access policies, fine-grained access control (index, document, field), Cognito or SAML for Dashboards, KMS at rest, TLS in transit and node-to-node.
• Ingestion: Firehose, IoT Core, CloudWatch Logs, OpenSearch Ingestion. Patterns: DynamoDB → Streams → Lambda → OpenSearch (search there, fetch from DynamoDB); CloudWatch Logs → subscription filter → Lambda (real time) or Firehose (near real time); KDS → Firehose (+ transform Lambda) or KDS → Lambda.
• Failures to remember: disk full blocks writes (use ISM and alarms), zero replicas turn a node loss into a red cluster, ignored REMOVE events leave ghosts, Lambda throttling delays logs, Firehose parks rejected documents in S3.
• New since 2023: OpenSearch Ingestion, zero-ETL from DynamoDB and direct query on S3 and CloudWatch Logs, vector search (k-NN) as the vector store for generative AI.

Self-check: Amazon OpenSearch Service32 questions