Kubernetes vs Nomad vs Bacalhau: Which Orchestrator is Right for Your Data?
If you are an engineer trying to manage petabytes of data scattered across different sources, you know the feeling associated with moving all of it to a central cloud cluster. It would take days–if not weeks–and cost a fortune in bandwidth fees.
Traditional orchestration tools like Kubernetes and Nomad have been the go-to solutions for managing workloads and have been enormously successful. But they are designed around a specific pattern–one where you move all your data next to your most powerful computer, usually in a single data center. However, data comes from everywhere, so this process can be slow, costly, and insecure for many use cases.
That’s where Bacalhau comes into play. Instead of moving data to compute, we bring compute to the data—whether it’s on edge devices, in the cloud, or on your own servers. The result? Faster processing, lower costs, and improved security.
This guide will help you understand:
- What Kubernetes, Nomad, and Bacalhau are
- The differences among Kubernetes, Nomad, and Bacalhau
- When to choose one of them, depending on your needs
- Why Bacalhau might be the missing piece in your infrastructure
Let's dive in!
Kubernetes, Nomad, Bacalhau: Core Concepts and Goals
Before making a comparison, it is important that you understand what these platforms are designed for and what goals they achieve.
Kubernetes
Kubernetes is among the most well-known and widely used container orchestrators. It is an industry standard and is offered as a hosted platform for Azure, AWS, and Google Cloud. It has a large ecosystem (Helm, Istio, Prometheus) and is primarily built for microservices that can communicate with each other, through internal mesh networking.
Nomad
Nomad is a lightweight orchestrator. It is simple, flexible, and reliable. It doesn’t try to do everything Kubernetes does. For example, while Kubernetes is primarily geared to running containers, Nomad is more open-ended and can run full VMs and standalone binaries. As it provides no internal networking, it also requires fewer services to get started, which can be simpler to build and maintain. Like Kubernetes, it is also multi-cloud friendly as it works across AWS, GCP, Azure, and on-prem.
Bacalhau
Bacalhau is designed to take the learning of Kubernetes and Nomad, and extend to the global network layout that many organizations face today. If you have ever asked the question "Why are we moving petabytes of data just to run a computation?", then Bacalhau is for you.
Here is the big idea behind Bacalhau: you can run compute jobs next to your data–whether it has just been generated, is sitting in an S3 bucket, stored in SQLite or an iceberg table, or streaming from edge devices. Bacalhau can also reuse the existing computer you have, rather than requiring new, completely dedicated servers, and saves you time and money. Because of this, Bacalhau helps reduce unnecessary data transfers, improving time to insights, security, and reducing costs. It does so without requiring you to adopt a new SDK or rewrite your applications, as it supports Docker, WASM, and other binaries.
In a few words, while Kubernetes and Nomad take a centralized approach and assume you will move data to compute, Bacalhau enables you to make the choice that fits your business better: move only what you need.
Comparing Kubernetes, Nomad, and Bacalhau: Key Differences
Despite its success, the traditional data stack presents major challenges:
Execution Environments
While all three platforms execute computational tasks, they differ in the types of runtimes they support natively and their primary focus:
- Kubernetes: It is primarily designed for container orchestration (supporting Docker and other runtimes) and offers an extensive ecosystem for managing containerized applications (like Helm and Operators). While support for WASM is emerging, its core strength remains in the container space. Also, note that managing diverse runtimes often requires additional tooling or custom extensions to the core API server.
- Nomad: Offers more flexibility in supported runtimes than Kubernetes. It natively orchestrates containers (Docker, Podman, and others) and non-containerized applications like virtual machines (VMs), static binaries, Java archives, and more. This makes it suitable for environments with a mix of legacy and modern workloads managed under a single system.
- Bacalhau: Bacalhau natively supports Docker containers and WASM as execution environments, including offering native pass-throughs of commands specific to those executions. Further, it offers a pluggable architecture that enables users to add their own custom binaries. This means it can run with any architecture, networking, or security constructs that are the right ones for that application. This includes the ability to run binaries or Docker containers that leverage specific hardware like GPUs for tasks such as CUDA programs, and taking full advantage of hardware acceleration.
Lightweight Execution and Mixed Architectures
The orchestrators differ in their resource footprint and their ability to manage workloads across diverse environments in the following terms:
- Kubernetes: Each cluster requires a dedicated control plane with multiple components (API server, etcd, scheduler). While powerful, this complexity can be heavy for simpler needs or resource-constrained environments. Its primary focus is orchestrating containers on relatively homogenous clusters, typically within a single cloud or data center. Kubernetes supports mixed architectures, but requires careful configuration and potentially specialized distributions, for example in the case of multi-platform builds.
- Nomad: Nomad runs as a single binary for both server and client nodes, making it lightweight and portable. Also, it can be deployed to different architectures based on your requirements. For example, you can use the constraint block to restrict the set of eligible nodes to particular architectures. In that case, the architecture attribute to specify is ${attr.cpu.arch}.
- Bacalhau: The Bacalhau agent is designed to be deployed to nearly any environment. Its focus is on executing your jobs on the nodes you provide, regardless of their execution type. This could be pre-deployed binaries, containers, WASM, or anything else that fits. This makes it suitable for mixed environments where the primary goal is distributed computation, not infrastructure management. In this case, the Architecture parameter specifies the architecture of the node.
Job Types
The three orchestrators manage different types of workloads, from tasks that run once to completion to long-running services. Their approaches vary, especially when dealing with distributed data:
- Kubernetes: It manages long-running services (via Deployments and StatefulSets) and handles batch processing through Job and CronJob resources. However, for large-scale distributed data processing, Kubernetes typically acts as the underlying infrastructure layer for specialized frameworks like Apache Spark or Flink, which manage the complex data computation logic across pods.
- Bacalhau: Bacalhau supports four different job types. Specifically, each job is, by default, called “embarrassingly parallel”–a fancy way of saying that the jobs do not need to (but can!) talk to each other to execute. That does not apply to every job, but to many jobs that can offer significant performance and processing speed-ups. Job types include batch (for tasks that run to completion on a specified number of nodes), service (for jobs that run continuously on a specified number of nodes), ops (for running jobs that are executed on all nodes that align with the job specification, but otherwise behave like batch jobs), and daemon (for jobs that run continuously on all nodes that meet the criteria given in the job specification.). Bacalhau’s goal is to give developers and data scientists the ability to execute in the way they need, rather than having the same job for everything.
Queued Execution
How jobs are handled when submitted versus when they actually run is another differentiator, especially for batch workloads:
- Kubernetes: Jobs submitted via Job or CronJob resources are queued by the Kubernetes scheduler if resources are not immediately available. Scheduling decisions are based on complex predicates and priorities. For large-scale batch systems running on Kubernetes (like Spark), those systems often implement their own more sophisticated queuing and scheduling logic on top of Kubernetes primitives.
- Nomad: Batch jobs submitted to Nomad are placed in evaluation queues. The Nomad scheduler continuously evaluates pending jobs and places them onto client nodes that satisfy the job's constraints as capacity becomes available.
- Bacalhau: Bacalhau implements an explicit job queuing mechanism that can be configured via a QueueTimeout setting. If enabled, a job will enter a Queued state if no suitable compute nodes are immediately available–either because all matching nodes are busy or because no nodes meet the job's specific requirements. The requester node will periodically retry assigning the job based on a QueueBackoff interval until the QueueTimeout expires.
Disconnected Execution
The ability to manage disconnections is different for the three platforms:
- Kubernetes: Kubernetes has different ways of managing disconnections and faults. For example, if one of the containers in the Pod fails, then Kubernetes may try to restart that specific container. However, Pods can fail in a way that the cluster cannot recover from. In those cases, Kubernetes deletes the Pod and relies on other components to provide automatic healing. At the node level, Kubernetes can manage graceful and non-graceful shutdowns. Also, Kubernetes is not natively designed for disconnected operations that can occur, for example, on edge devices due to network instability
- Nomad: Nomad clients are designed to be resilient to temporary network partitions. Clients can continue running their allocated tasks even if disconnected by using the disconnect block. It also natively supports edge computing while, however, maintaining a centralized approach.
- Bacalhau: Bacalhau is designed with distributed and potentially disconnected or intermittently connected environments in mind, like edge environments. In particular, it provides a robust execution framework that allows task execution without requiring persistent network connections to a central controller. Jobs can be told to fail immediately, or queued for a determined amount of time to wait for the resources to reconnect. This ensures secure and reliable computing at the edge.
Security
While all orchestration platforms offer security features, their architectures significantly impact data security posture, particularly concerning the risk of data exposure and leakage:
- Kubernetes and Nomad: They operate on a centralized model where data is typically moved to the compute cluster, or the cluster needs broad access to potentially disparate data sources. While they provide robust security mechanisms like Kubernetes' RBAC, Network Policies, and Secrets, or Nomad's ACLs and integrations like Consul, the centralized model creates inherent risks such as metadata leakage. Centralized systems often aggregate logs, credentials, and job configuration details. Even if the primary data is not leaked, the leakage of this operational metadata can reveal sensitive information about the data, computations, or infrastructure, potentially providing attackers with valuable intelligence or credentials.
- Bacalhau: Bacalhau's security advantage lies directly in its core "Compute Over Data" principle that minimizes data movement. By executing computations directly where the data resides, Bacalhau reduces the need to transfer large, potentially sensitive datasets across networks, without requiring a significant re-architecture. Also, since not all job metadata is aggregated centrally, the risk of accidental breaches exposing data is significantly lower.
Kubernetes, Nomad, and Bacalhau: Table of Comparison
Choosing Your Orchestrator: Kubernetes vs Nomad vs Bacalhau Use Cases
Choosing between Kubernetes, Nomad, and Bacalhau depends heavily on your specific needs, infrastructure, team expertise, and the nature of your workloads, especially concerning data.
In this last section, you will read a breakdown with specific use cases to help you make an informed choice.
When To Choose Kubernetes
Kubernetes is the industry standard for container orchestration, particularly strong for managing complex, scalable, containerized applications, especially microservices.
It is your best choice when you need to run complex microservices or interconnected microservices at scale, in a single location. If your application consists of many independent, containerized services that need sophisticated networking, service discovery, auto-scaling, rolling updates, and self-healing, Kubernetes provides the most comprehensive feature set. For example, a large e-commerce platform with dozens of microservices for product catalogs, user accounts, shopping cart.
When To Choose Nomad
Nomad focuses on simplicity and flexibility. It is capable of orchestrating containers but also non-containerized applications like VMs, Java applications, or raw executables.
Nomad is best when you need simpler, high-scale orchestration inside a single zone. If Kubernetes feels overly complex for your needs and you value a simpler architecture, Nomad is a strong contender. For example, a medium-sized company in need of reliable scheduling for a mix of web services and batch jobs without the full Kubernetes overhead can be the right fit to be orchestrated by Nomad.
Additionally, if Kubernetes is too restrictive in the zone, and you need more job types than just Docker or WASM, Nomad may also be a fit. If your environment includes not just Docker containers but also virtual machines, standalone binaries, or Java applications, Nomad can manage them all natively within a single cluster and tool. For example, an organization migrating legacy applications alongside newer containerized ones, wanting a single orchestrator, may choose Nomad.
When To Choose Bacalhau
Bacalhau specializes in Compute Over Data, focusing on running computations where large datasets reside, particularly in distributed or edge environments. Here is when you should choose it:
- Processing distributed datasets is your primary goal: If moving your data to a central compute cluster is impractical, costly, or prohibited due to size, bandwidth limitations, or regulations, Bacalhau is designed for this. For example, analyzing petabytes of satellite imagery stored across different regional archives without transferring it all is a good example where Bacalhau shows its superpowers at your disposal.
- Data gravity or sovereignty is a major concern: When data cannot leave a specific geographic location or administrative domain due to regulations (like GDPR) or organizational policy. A typical example can be performing analysis on sensitive medical data held by different hospitals without pooling the raw data.
- Edge computing requires processing data at the source: Running analytics or ML inference directly on edge devices or local gateways where data is generated. A use-case example can be processing sensor data from industrial equipment on-site for real-time anomaly detection.
Conclusion
Choosing the right orchestrator is not about finding the single "best" tool, but the best fit for your specific challenges. Kubernetes remains the powerhouse for complex, cloud-native container orchestration at scale. Nomad, on the other hand, offers a compelling blend of simplicity and flexibility for managing diverse workloads, especially within the HashiCorp ecosystem.
However, if your primary bottleneck involves processing massive, distributed datasets where data movement is costly, slow, or restricted, Bacalhau is your best choice. By embracing the Compute Over Data model, Bacalhau directly tackles the challenges of data gravity, offering savings in time and cost, alongside security benefits from data locality and WASM sandboxing.
What's Next?
To start using Bacalhau, install Bacalhau and give it a shot.
However, if you don’t have a network and you would still like to try it out, we recommend using Expanso Cloud. Also, if you would like to set up a cluster on your own, you can do that too (we have setup guides for AWS, GCP, Azure, and many more 🙂).
Get Involved!
We welcome your involvement in Bacalhau. There are many ways to contribute, and we’d love to hear from you. Please reach out to us at any of the following locations:
- Expanso’s Website
- Bacalhau’s Website
- Bacalhau’s Bluesky
- Bacalhau’s Twitter
- Expanso’s Twitter
- TikTok
- Youtube
- Slack
- Careers Page
Commercial Support
While Bacalhau is open-source software, the Bacalhau binaries go through the security, verification, and signing build process lovingly crafted by Expanso. You can read more about the difference between open-source Bacalhau and commercially supported Bacalhau in our FAQ. If you would like to use our pre-built binaries and receive commercial support, please contact us or get your license on Expanso Cloud!
Ready to get started?
Create an account instantly to get started or contact us to design a custom package for your business.
