Splunk Architecture Explained: Components, Data Flow, and Optimization

Most teams can name Splunk’s components: forwarders, indexers, search heads. They can draw the standard three-tier architecture of Splunk on a whiteboard from memory. But the architecture layer that determines how much data actually reaches Splunk (and what it costs) is absent from nearly every diagram. That gap between data sources and Splunk ingestion is where the real optimization happens. This guide covers the full Splunk architecture, component by component, then addresses the part most people skip. If you are evaluating your data observability and monitoring stack, understanding this architecture is essential.

Key Takeaways

Architecture determines cost: Splunk licensing is volume-based, so how you design data ingestion directly controls your bill. Getting architecture right is not optional: it is the single biggest lever for managing Splunk spend.
Seven core components work together: Universal Forwarders, Heavy Forwarders, Indexers, Search Heads, Deployment Server, Cluster Manager, and License Manager each play a distinct role. Understanding what each one does, and does not do, prevents misconfigurations that cause outages and data loss.
The missing layer is upstream filtering: Most Splunk deployments push everything into the indexing tier and sort it out later. Adding a processing layer before data reaches Splunk lets you reduce volume, enrich events, and route data intelligently, cutting costs without losing visibility.

Splunk Architecture Overview

Splunk is a platform for searching, monitoring, and analyzing machine-generated data. At its core, the architecture follows a pipeline: data comes in, gets parsed and indexed, and then becomes searchable. Every Splunk deployment, whether a single-server lab setup or a multi-site enterprise cluster, follows this same basic pattern.

The architecture can be broken into three tiers:

Data Collection Tier: Forwarders collect data from sources and send it downstream
Indexing Tier: Indexers parse, store, and make data searchable
Search Tier: Search Heads provide the interface for users to query indexed data

Between and around these tiers sit management components that handle configuration distribution, cluster coordination, and license tracking. The size of your deployment determines which components you need and how many instances of each.

A minimal deployment might run everything on a single server. A production deployment for a mid-size enterprise typically involves dozens of forwarders, several indexers, a search head cluster, and dedicated management nodes. Large enterprises can run hundreds of indexers processing terabytes per day.

The key thing to understand: Splunk’s architecture is modular. Each component can be scaled independently, which gives you flexibility but also means there are many ways to get the design wrong.

Core Components of Splunk Architecture

Universal Forwarder (UF)

The Universal Forwarder is the lightweight agent you install on every machine that generates data you want in Splunk. It has a small footprint, typically consuming minimal CPU and memory, because its job is simple: read data from files or inputs and forward it to the next tier.

It collects data from log files, Windows Event Logs, metrics and other local sources, and sends it to indexers or Heavy Forwarders over TCP, typically on port 9997. What it sends is Splunk’s cooked format, streamed unparsed rather than as raw bytes; forwarding genuinely raw data is a separate setting. It buffers in memory while a downstream receiver is unavailable, and supported inputs can be configured with persistent queues on disk, but neither is unlimited: once the queue fills, the forwarder blocks its inputs, and a blocked input on a source that does not wait, such as a UDP listener, drops events. It also load-balances across several receivers to spread the data evenly.

What it does not do is just as deliberate. It forwards data unparsed, leaving timestamp extraction, field extraction and content-based routing to the tier above it, and it does not index anything locally, run searches or offer a web interface. Two exceptions are worth knowing: structured files such as CSV are handled at the forwarder, and EVENT_BREAKER_ENABLE with EVENT_BREAKER lets it set event boundaries before sending, which matters when you are load-balancing a single large source across several indexers.

This is intentional. By keeping the forwarder lightweight, you can install it across thousands of endpoints without worrying about resource impact. The parsing and heavy lifting happens downstream.

Configuration: UFs are configured via inputs.conf (what to collect) and outputs.conf (where to send it). In large deployments, these configurations are pushed from a Deployment Server rather than managed individually on each machine.

Heavy Forwarder (HF)

The Heavy Forwarder is a full Splunk instance configured to forward data rather than index it locally. Unlike the Universal Forwarder, the Heavy Forwarder can parse, filter, and route data before sending it downstream.

Because it runs the full Splunk processing pipeline, it parses at the forwarder level, applying transforms, event breaking and timestamp extraction before anything moves on. It filters and routes through transforms.conf and props.conf, dropping unwanted events, masking sensitive fields, or sending specific sourcetypes to specific indexers. It also reaches sources a Universal Forwarder cannot, including APIs, databases through DB Connect, and message queues.

When to use a Heavy Forwarder:

You need to collect from sources that require a full Splunk instance (syslog over TCP/UDP, scripted inputs with complex dependencies, API-based collection)
You want to filter or transform data before it reaches the indexing tier to reduce volume
You need to route data to different indexers or indexes based on content
Compliance requires masking sensitive data before it leaves a specific network zone

The tradeoff is resource consumption. A Heavy Forwarder needs significantly more CPU and memory than a Universal Forwarder because it runs the full parsing pipeline. In a well-designed architecture, you use UFs wherever possible and HFs only where their additional capabilities are required.

Indexer

The Indexer is the workhorse of Splunk. It receives data from forwarders, parses it into individual events, stores it in indexes, and serves search requests against that data.

It parses incoming data, breaking raw streams into individual events, extracting timestamps and applying line breaking rules, then transforms it with the field extractions, lookups and event modifications defined in props.conf and transforms.conf. It writes the result to index buckets on disk, organised by time and index name. When a Search Head sends a request, the Indexer runs that search against its local data and returns the results, and it moves data through its lifecycle as retention policies require.

Index storage works in buckets:

Hot buckets: actively being written to, stored on fast storage
Warm buckets: recently closed hot buckets, still on primary storage
Cold buckets: older data, can be moved to cheaper storage
Frozen buckets: data that has aged out of retention; by default deleted, but can be archived

In a clustered environment, Indexers replicate data to other Indexers to provide redundancy. The replication factor determines how many copies of raw data exist, and the search factor determines how many searchable copies exist.

Sizing: Indexer capacity is measured in gigabytes of data ingested per day. A common rule of thumb is that one Indexer can handle roughly 100-300 GB/day depending on hardware, search load, and data complexity. This varies significantly based on the type of data and the search patterns against it.

Search Head

The Search Head is the user-facing component. It provides the web interface, runs searches, and presents results. In a distributed deployment, the Search Head does not store indexed data: it dispatches search jobs to Indexers and aggregates the results.

Everything a user sees runs here: dashboards, the search bar, reports, alerts and apps. It breaks each search into parts, sends them to the Indexers, then merges the partial results it gets back. Scheduled work runs here too, including alerts, reports and summary indexing searches, and the Search Head stores the knowledge objects behind all of it, from saved searches and field extractions to tags, event types and dashboards.

In a Search Head Cluster (SHC), multiple Search Heads share knowledge objects and distribute the search workload. An SHC requires a minimum of three members and a deployer to push apps and configurations to the cluster members.

Key consideration: The Search Head is often the bottleneck in Splunk deployments. Complex dashboards, concurrent users, and expensive searches can overwhelm Search Head CPU and memory. Monitoring Search Head performance and managing search concurrency is critical for a healthy deployment.

Deployment Server

The Deployment Server is the configuration management hub for forwarders. It pushes apps and configuration updates to Universal Forwarders and Heavy Forwarders, letting you manage thousands of forwarders from a single point.

Forwarders register with it and check in periodically. It groups them into server classes by criteria such as hostname, IP range or operating system, assigns configuration bundles to those classes, pushes them to whichever forwarders match, and can trigger a restart once a change lands.

Its reach is wider than forwarders alone: it can also push apps and configuration to non-clustered indexers and standalone search heads. Where it stops is clustering. Indexer-cluster peer nodes take their configuration bundles from the Cluster Manager, and search-head-cluster members from the SHC Deployer, precisely so a cluster stays internally consistent. It reports which clients have checked in, which tells you a forwarder is alive but not whether it is healthy; that is the Monitoring Console’s job. And it is not a configuration management tool for the wider Splunk environment.

Important limitation: The Deployment Server is a single point that can become a bottleneck at scale. Deployments with more than a few thousand forwarders may experience slow check-ins and configuration propagation delays. Planning forwarder check-in intervals and staggering deployments helps, but very large environments may need to consider multiple Deployment Servers or alternative configuration management approaches.

Cluster Manager (formerly Master Node)

The Cluster Manager coordinates the Indexer Cluster. It manages bucket replication, handles peer node membership, and ensures the cluster meets its configured replication and search factors.

What the Cluster Manager does:

Manages cluster membership: Indexer peers register with the Cluster Manager
Coordinates bucket replication: ensures the correct number of copies of each bucket exist across the cluster
Handles peer failures: when an Indexer goes down, the Cluster Manager directs other peers to replicate the missing copies
Pushes configuration bundles: distributes index-time configurations to all Indexer peers
Manages rolling restarts: coordinates restarts across the cluster to maintain availability

The Cluster Manager does not process or store data itself. It is a coordination node. However, losing the Cluster Manager is a serious event: while existing data remains searchable, new data cannot be properly replicated and the cluster cannot perform maintenance operations until the Cluster Manager is restored.

High availability: In recent Splunk versions, you can configure a standby Cluster Manager for failover. This is strongly recommended for production environments where cluster management downtime is not acceptable.

License Manager

The License Manager tracks how much data is being indexed across your Splunk deployment and enforces licensing limits.

What the License Manager does:

Tracks daily ingestion volume: every Indexer reports its daily indexing volume to the License Manager
Enforces license limits: if you exceed your licensed volume, Splunk issues warnings and eventually restricts search functionality
Manages license pools: allows you to allocate license capacity across different groups of Indexers
Provides usage reports: helps you understand ingestion trends and plan capacity

License violations: Splunk allows you to exceed your daily licensed volume up to five times in a rolling 30-day window before triggering a violation. A violation blocks search for non-admin users until the violation is resolved. Monitoring license usage trends and setting alerts at 80-90% of your licensed volume is a best practice that prevents surprises.

In smaller deployments, the License Manager role can run on an existing Splunk instance (often the Search Head or Cluster Manager). In larger deployments, it is typically a dedicated instance for reliability.

Splunk Cloud vs. Splunk Enterprise Architecture

The choice between Splunk Cloud and Splunk Enterprise (on-premises) fundamentally changes your architecture responsibilities.

Splunk Enterprise (On-Premises)

With Splunk Enterprise, you own the entire stack:

You manage all components: Indexers, Search Heads, Cluster Manager, License Manager
You handle capacity planning, hardware procurement, OS patching, and Splunk upgrades
You control data locality, network security, and storage tiering
You are responsible for high availability, disaster recovery, and backup strategies

This gives you maximum control but also maximum operational burden. A well-run on-premises Splunk environment requires dedicated admin staff.

Splunk Cloud

With Splunk Cloud, Splunk manages the indexing and search tiers:

Splunk manages Indexers, Search Heads, Cluster Manager, and the underlying infrastructure
You manage data collection (forwarders) and data inputs
You configure knowledge objects, apps, dashboards, and alerts through the Splunk Cloud UI or API
Splunk handles upgrades, scaling, and availability of the backend

You still need to deploy and manage Universal Forwarders and any Heavy Forwarders on your end. Data collection architecture is still your responsibility. The difference is that once data reaches Splunk Cloud, the infrastructure management is handled for you.

Hybrid Architectures

Many organizations run hybrid architectures:

Splunk Cloud for primary SIEM: security data goes to Splunk Cloud for the managed experience
On-premises Splunk for sensitive data: data that cannot leave the network stays on local Indexers
Federated search: Search Heads can query across both Cloud and on-premises Indexers

Hybrid architectures add complexity but can be the right choice when you have data sovereignty requirements or workloads that benefit from local processing.

Splunk SIEM Architecture

When Splunk is deployed as a Security Information and Event Management (SIEM) platform, the base architecture remains the same but the design priorities shift.

Data Sources for SIEM

A Splunk SIEM deployment typically ingests:

Network security: Firewall logs, IDS/IPS alerts, proxy logs, DNS logs, NetFlow
Endpoint security: EDR telemetry, antivirus logs, sysmon events, Windows Security Event Logs
Identity and access: Active Directory logs, LDAP, VPN logs, MFA logs, SSO events
Cloud and SaaS: AWS CloudTrail, Azure Activity Logs, GCP Audit Logs, Office 365, Okta
Application security: Web application firewall logs, authentication logs, database audit logs

SIEM-Specific Architecture Considerations

Volume is the challenge: Security data sources are high volume. A single firewall can produce 50-100 GB/day. Multiply that across a perimeter with dozens of firewalls, and the SIEM data alone can dominate total Splunk license consumption. Endpoint telemetry from thousands of machines adds up fast. SIEM deployments often represent the largest Splunk environments in an organization.

Correlation drives value: The power of Splunk as a SIEM comes from correlating events across sources. This means you need all relevant data in the same Splunk environment and properly normalized so that correlation searches work effectively.

Enterprise Security (ES): Splunk’s premium SIEM app runs on top of the base platform. It adds a Notable Events framework for alert triage, Risk-Based Alerting to reduce alert fatigue, and a threat intelligence framework for indicator matching, along with pre-built correlation searches and dashboards and the compliance reporting and investigation workbenches a SOC needs.

ES is resource-intensive. It runs many scheduled searches and summary indexing jobs, so Search Head capacity needs to be sized accordingly.

SOAR integration: Splunk SOAR (Security Orchestration, Automation, and Response) connects to the SIEM to automate response playbooks. Events trigger playbooks that can contain, investigate, and remediate threats without manual intervention. SOAR is a separate deployment but tightly integrated with the Splunk SIEM architecture.

Tiered Storage for SIEM

Security data often has long retention requirements (90 days, 1 year, or more for compliance). Splunk’s SmartStore and index tiering help manage cost:

Hot/Warm storage on fast SSDs for recent, actively searched data
SmartStore with remote object storage (S3, GCS, Azure Blob) for warm data that is accessed less frequently
Cold/Frozen tiers for long-term retention on cheaper storage

Designing storage tiers correctly is critical for SIEM deployments because the combination of high volume and long retention creates massive storage requirements.

The Missing Layer: What Happens Before Splunk

Open any Splunk architecture diagram. It starts at the forwarder. Data sources on the left, forwarders in the middle, indexers on the right. Clean, simple, familiar.

But what happens between the data source and the forwarder? That space is the highest-impact optimization surface in the entire architecture, and it is almost always blank. This is where data governance practices should begin – at the source, before data enters any downstream system.

Here is the reality: 60-70% of machine data generated by infrastructure, applications, and security tools adds no value to the searches, dashboards, and alerts that justify Splunk’s cost. Debug logs that nobody queries, duplicate events from redundant collection paths, verbose health checks that fire every 5 seconds, telemetry fields that no correlation search references. It all gets indexed. It all counts against the license.

Splunk’s own Heavy Forwarder can filter and route. But HFs are full Splunk instances. They cost resources, require their own management layer, and operate after data has already traversed the network. They are a forwarding tool, not an edge compute platform.

The alternative is moving filtering, transformation, and routing to the data source itself. An agent that sits at the origin, applies rules to drop noise, redact sensitive fields, convert verbose logs into compact metrics, and route only the relevant data forward to Splunk. Learn more about the Expanso product and how it fits into this architecture.

One global bank did exactly this. Their Splunk environment ingested 14.3 TB/day across security, infrastructure, and application data. After deploying lightweight agents at the source that filtered known-noise event types, deduplicated redundant log streams, and converted verbose application traces into summary metrics, daily ingestion dropped to 5.2 TB/day, a 64% reduction. Annual Splunk costs went from $3.7M to $1.4M, a 62% cut. Search performance improved by 4x because indexers processed less data per query. The full details are in the Splunk cost optimization case study.

The agents handling this were Expanso’s Compute Over Data platform, running a lightweight agent per node, with a native Splunk HEC output connector.

The point is not that Splunk is overpriced. The point is that every architecture diagram should include what happens before the forwarder, because that layer determines the cost and performance of everything after it.

What an Upstream Processing Layer Does

An upstream processing layer sits between your data sources and Splunk, intercepting data after collection but before indexing.

It drops events with no analytical value, such as debug logs in production, repetitive health checks and known noise, and samples high-volume low-value sources rather than indexing every event. It aggregates where the detail does not earn its keep, rolling a thousand identical firewall allows into one count per source and destination pair. It enriches events with GeoIP, asset lookups and threat intelligence tags before indexing, so Splunk is not doing that work at search time, and normalises field names and formats so correlation searches run without complex search-time extractions. Routing is the last step: high-value security events to Splunk, low-value operational logs to cheaper storage, by content, value or compliance requirement.

The implementation is light: one agent runs at each source, with a native Splunk HEC output connector and 180+ components to build the rest of the pipeline from. Configurations propagate to 8,500+ nodes in <30 seconds, and the same control plane has been tested at 100,000+ nodes.

Optimizing Splunk Architecture for Cost and Performance

Optimization within Splunk falls into four categories. Each helps. None fully solves the volume problem.

Index Optimization

Index segmentation. Separate indexes by source type, retention requirement, and access pattern. Security data that requires 1 year of hot/warm storage should not share an index with application debug logs that expire after 30 days.
Volume indexes. Use volume settings in indexes.conf to enforce disk quotas per index, preventing a noisy source from consuming all available storage.
Bucket sizing. Tune maxDataSize to balance search performance (smaller buckets = more files to open) against indexing throughput (larger buckets = fewer writes).

Search Optimization

Time-bound every search. A search without earliest and latest scans every bucket. Set explicit time ranges.
Use indexed fields. Fields extracted at index time (source, sourcetype, host, index) are faster to filter on than search-time extractions.
Replace wildcards. Leading wildcards (*error*) force full scans. Use TERM() or PREFIX() directives when possible.
Accelerate reports. Summary indexing and report acceleration precompute results for dashboards that run the same expensive searches repeatedly.

Data Model Tuning

For ES deployments, data model acceleration consumes significant disk and CPU. Tune acceleration by:

Limiting the time range of accelerated data to the minimum your dashboards and correlation searches require.
Constraining data model searches with where clauses to reduce the events accelerated.
Disabling acceleration on data models you do not actively use.

Retention Policies

Set retention per index based on actual use. Most teams apply blanket retention policies (90 days, 1 year) across all indexes, but different data types have different value curves. Teams rarely query firewall connection logs older than 30 days. Audit logs may need 7 years. Align retention to query patterns, not compliance defaults applied uniformly.

The Ceiling: Why Internal Optimization Has Limits

All four categories above operate on data that Splunk already indexed. They reduce search cost and storage cost, but they do not reduce license cost. The license meter ticks at ingestion, before any of these optimizations apply.

This is why upstream filtering is the ceiling-breaker. Reducing data before it reaches the indexing tier is the only optimization that impacts license, storage, and search performance simultaneously.

Reference Architecture: Splunk + Upstream Filtering

Here is the architecture pattern that addresses the volume problem at its source:

Data Sources (servers, network devices, cloud services, containers)
    |
    v
Upstream Agents (filter, transform, enrich, route at the edge)
    |
    v
Splunk HEC / Forwarder Input (only relevant, governed data)
    |
    v
Indexer Cluster (less volume = faster indexing, smaller buckets)
    |
    v
Search Head Cluster (faster searches across leaner indexes)

How the upstream layer works:

Filter at the source. Drop events that match known-noise patterns: health checks, debug traces, duplicate heartbeats. Rules are defined centrally and pushed to edge agents.
Transform before sending. Convert verbose multi-line stack traces into single-line summaries. Aggregate high-frequency metrics into periodic rollups. Redact PII fields before data leaves the network boundary.
Route by value. High-value security events go to Splunk via HEC. Low-value operational telemetry routes to cheaper object storage for compliance retention. Medium-value data goes to a metrics store. One data stream, three destinations, each at the right cost.
Govern at the edge. Enforce data residency rules before data moves. Mask sensitive fields at the source, not in a downstream pipeline that has already transported unmasked data across zones.

A telecommunications provider with 3,847 cell sites used this pattern. Each site ran lightweight upstream agents that applied telemetry filtering rules at the edge. The result: 78% reduction in telemetry volume reaching the central Splunk deployment, a 47% drop in Splunk costs, and local compliance with data residency requirements across 12 countries.

The fleet management aspect matters at scale. Expanso supports configuring 8,500+ nodes in under 30 seconds and has handled 100,000+ nodes in testing, with 180+ components including a native Splunk HEC output connector. For enterprise Splunk deployments with thousands of forwarders, the upstream layer deploys and updates as a fleet, not a one-node-at-a-time manual process.

An enterprise IT team running this architecture saw monthly Splunk costs drop from $240K to $71K (a 70% reduction) while query latency improved from 45 seconds to 2.8 seconds, a 16x speedup. Less data in the index means faster searches across the board.

Production Reference Components

A production-grade Splunk Enterprise deployment for a mid-to-large organization typically looks like this:

Data Collection Tier

Universal Forwarders on all servers, endpoints, and network devices that support agents
Heavy Forwarders for syslog aggregation, API-based data collection, and data filtering/routing
Upstream processing layer (such as Expanso) for filtering, enrichment, and routing before data reaches the indexing tier

Indexing Tier

Indexer Cluster with a minimum of 3 Indexers (production environments typically run 6+)
Replication factor of 3 for data durability (three copies of raw data)
Search factor of 2 for search availability (two searchable copies)
SmartStore with object storage for warm data cost optimization
Cluster Manager (plus standby) for cluster coordination

Search Tier

Search Head Cluster with a minimum of 3 members for high availability
SHC Deployer for pushing apps and configurations to the cluster
Dedicated ES Search Head if running Enterprise Security (ES is resource-intensive and benefits from isolation)

Management Tier

Deployment Server for forwarder configuration management
License Manager (can run on the Cluster Manager in smaller deployments)
Monitoring Console for infrastructure health visibility

Network Considerations

Forwarders -> Indexers: TCP 9997 (data forwarding), SSL recommended
Search Heads -> Indexers: TCP 8089 (management), TCP 9997 (data)
All components -> Cluster Manager: TCP 8089
All components -> License Manager: TCP 8089
Users -> Search Heads: TCP 443 (web UI, HTTPS)

Firewall rules should allow these flows while restricting unnecessary lateral communication. In multi-site deployments, consider bandwidth between sites for replication traffic.

Closing Thought

Splunk architecture is well-documented at the component level. What’s less documented is the gap between your data sources and your forwarders: the space where data volume, cost, and search performance are actually determined. Whether you optimize internally with Heavy Forwarder filtering and index tuning, or add an upstream compute layer that processes data before it ever reaches the forwarder, the goal is the same: make sure every byte Splunk indexes is a byte worth indexing.

If you’d like to see how this applies to your environment, talk to an engineer about reducing your Splunk data volume.

FAQ

What are the main components of Splunk architecture?

Splunk architecture consists of three tiers. The data input tier includes Universal Forwarders, Heavy Forwarders, and HEC endpoints that collect and send data. The indexing tier uses Indexer clusters to parse, store, and replicate events. The search tier uses Search Head clusters to run queries, dashboards, and alerts. Supporting components include the Deployment Server (forwarder configuration), Cluster Manager (indexer coordination), and License Manager (ingestion tracking).

How does Splunk Cloud architecture differ from Splunk Enterprise?

Splunk Cloud runs the same three-tier architecture but manages the indexing and search tiers for you. You do not provision or configure indexers or search heads at the OS level. Data collection still uses on-prem Universal Forwarders managed by your Deployment Server, alongside cloud-native inputs. The pricing model shifts entirely to ingestion volume, making upstream data reduction even more impactful.

Why is Splunk SIEM (Enterprise Security) so expensive to run?

ES adds data model acceleration, continuous correlation searches, and notable event generation on top of base Splunk. Security data sources (firewalls, DNS, endpoint agents, authentication systems) produce the highest event volumes in most environments. Data model acceleration writes additional index files. Correlation searches run on tight schedules. The combination of high volume, additional indexing overhead, and continuous search load makes ES the most resource-intensive Splunk workload.

What is the difference between a Universal Forwarder and a Heavy Forwarder?

A Universal Forwarder is a lightweight agent that collects and sends raw data without parsing. It uses minimal resources and targets mass deployment. A Heavy Forwarder is a full Splunk instance that can parse, filter, mask, and route data before forwarding. Heavy Forwarders require more CPU and memory and are typically used for data transformation tasks like PII masking or conditional routing between multiple indexer clusters.

How do I reduce Splunk license costs without losing visibility?

Start with index segmentation and retention tuning to reduce storage costs. Optimize searches with time bounds and indexed field filters to reduce compute costs. For license cost reduction, you must reduce ingestion volume. Internal options include filtering via Heavy Forwarders and disabling unnecessary inputs. The highest-impact approach is upstream filtering at the data source, which removes noise before it enters the Splunk pipeline, reducing license, storage, and search costs simultaneously.

Can Splunk handle multi-site deployments?

Yes. Splunk’s indexer clustering supports multi-site configurations where you define site-aware replication and search factors. For example, site_replication_factor = origin:2, total:3 ensures two copies stay at the originating site and three copies exist across all sites. Search affinity directs search heads to prefer local indexers, reducing cross-site network traffic. The Cluster Manager coordinates all of this from a central location.

What is SmartStore in Splunk?

SmartStore is a storage architecture that separates compute (Indexers) from storage by using remote object storage (like Amazon S3, Google Cloud Storage, or Azure Blob Storage) for warm bucket data. Indexers keep hot data locally for write performance and cache frequently accessed warm data, but the primary copy of warm data lives in object storage. This reduces the local storage requirements on Indexers and allows storage to scale independently of compute. SmartStore is particularly valuable for large deployments with long retention requirements.

How many Indexers do I need?

Indexer count depends on your daily ingestion volume, search load, data retention period, and hardware specifications. A common guideline is that one Indexer can handle 100-300 GB/day, but this varies significantly based on data complexity, search concurrency, and hardware. For a clustered deployment with a replication factor of 3, you need a minimum of 3 Indexers. Production environments handling 500 GB/day or more typically run 6 or more Indexers. Always monitor Indexer queue health and performance metrics to determine when to add capacity.

What happens if the Cluster Manager goes down?

If the Cluster Manager becomes unavailable, existing indexed data remains searchable and Indexers continue to receive and index new data. However, bucket replication cannot be coordinated, new peers cannot join the cluster, rolling restarts cannot be performed, and configuration bundle pushes are blocked. If an Indexer fails while the Cluster Manager is down, the missing bucket copies will not be replicated to other peers until the Cluster Manager is restored. For production environments, configuring a standby Cluster Manager for failover is strongly recommended.

What is the role of an upstream processing layer in Splunk architecture?

An upstream processing layer sits between data sources and Splunk to process data before indexing. It performs filtering (dropping low-value events), sampling (indexing a percentage of high-volume sources), aggregation (combining similar events into summaries), enrichment (adding context like GeoIP or asset information), routing (sending data to different destinations based on value), and normalization (standardizing formats for easier correlation). This layer reduces the volume that reaches Splunk, lowers licensing costs, improves search performance, and enables smarter data management without sacrificing collection completeness.