During an incident it costs the most and helps the least
Six problems, and the first two are what usually starts the conversation.
The observability bill grows faster than the estate — fastest of all during an incident, when everyone logs more — and nobody can say which team or service is responsible for which part of it.
Growing, unattributedEverything pages, so nothing means anything. The on-call engineer has learned which alerts to ignore, which is the most dangerous state a system can be in, because it looks like monitoring.
Learned to ignoreIncidents start with archaeology: the first ten minutes go on finding where the logs are rather than on reading them.
Ten minutes goneThe least reliable thing in production — sized by guesswork two years ago, and it fails hardest during incidents, which is exactly when it is needed.
Fails when neededIt cannot answer the question that was asked: either the logs are gone before the auditor’s window, or everything is kept hot forever because nobody set a lifecycle policy.
Wrong at both endsMetrics and logs do not correlate. The spike is on one system and the explanation is on another, with different timestamps, different labels and no link between them.
No link between themNothing central, or too much of it — both arrive here
Two readers, at opposite ends of the same problem.
Nothing central.
Debugging means connecting to a box, or guessing which box. Metrics are whatever the cloud provider shows by default. Nobody decided this was a problem — an incident decided it for them.
Too much, badly.
There is a stack. The bill doubled year on year, or the cluster nobody sized has become the least reliable thing in production, or everything alerts and therefore nothing does. This reader is not asking what observability is; they are asking how to make theirs cost less and mean more.
We make the signal exist. What is worth watching is a different job.
Observability
Answers what should we be watching — service levels, error budgets, what is worth instrumenting and why.
The stack
Makes the signal exist and cost the right amount. It answers where are the logs.
Instrumentation and tracing
We do the plumbing — collectors, pipelines, storage. Deciding what to instrument and which user journeys matter belongs to the practice.
You can buy the stack without the practice, and plenty of teams should.
Builds and rescues — and most people arrive for a rescue
Assess and select
The most valuable hour on this page. What is actually being logged and at what volume, what it costs today, what the retention obligation really is, where the cardinality is coming from, and what your team can realistically operate. You get a written recommendation with a stack chosen for that estate and a cost projection — not a preference we brought with us.
Build the logging pipeline
Collection running as a DaemonSet on every node, into a backend chosen for the estate. Parsing and structure so logs are queryable rather than merely stored. Index lifecycle and retention tiering so hot, warm and cold data cost what they should. A query and dashboard layer your team will actually open, with role-based access control wired to your existing identity provider — so giving a developer log access is not a production risk.
Build metrics and alerting
Collection and storage sized to the estate, dashboards somebody will actually open, and routing, grouping and silences that hold up during an incident. The part everyone skimps: alert rules that correspond to something a human should do. Every rule we write has a runbook entry, or it does not get written — an alert that fires and needs no action trains the on-call engineer to ignore the next one.
Fix what exists
The cost and noise engagement, and the one most people arrive for. Retention tiering, sampling, dropping the labels behind a cardinality explosion, moving to a different engine where that is genuinely the answer, consolidating three overlapping tools, and cutting alert volume by rewriting rules against what actually happened rather than what someone once feared might.
The selection is the work
A consultancy that always recommends the same stack is describing its own comfort rather than your problem. Here is what decides it, and when each one is the right answer.
Log storage and search
Elastic stack
Elasticsearch, Fluent Bit and Kibana — where log search and analysis genuinely matter: full-text query, mature access control, and people who live in Kibana all day. Fluent Bit rather than Logstash, for a plain reason: its footprint is small enough to run on every node without a memory argument, which is why it displaced Logstash in Kubernetes estates.
Loki
A Kubernetes-shaped estate where label-based lookup is enough, at a fraction of the storage cost.
Metrics and alerting
Prometheus, Grafana, AlertManager
The metrics and alerting default.
When one of them hits a wall
VictoriaLogs and VictoriaMetrics
Where Elasticsearch or Prometheus have hit a scale or cost wall and the answer is a different engine rather than a bigger cluster. Usually the second conversation, and we would rather have it before the invoice does.
Instrumentation and traces
OpenTelemetry
Instrumentation, vendor-neutral, so the backend decision stays reversible.
Grafana Tempo
Traces.
It runs in your infrastructure, and none of it stays with us
Self-hosted
In your own infrastructure, or on infrastructure we run for you. No SaaS seat to keep paying for, and no per-gigabyte meter running against you.
Or the cloud provider’s own
CloudWatch, Azure Monitor or Google Cloud Operations, where the estate suits it. For a small single-cloud estate that is genuinely the right answer, and pretending otherwise sells work nobody needs.
Deployed like everything else
As code, through a pipeline, in your repositories — the same way we build the rest of an estate.
Exit
Everything lives in your own repositories and your own cloud accounts. End the engagement and you keep all of it; another team can pick it up.
Cloud billing
You pay your cloud costs directly. We are not between you and your provider, so you keep your own commitments, discounts and negotiated rates. Cloud spend is not part of the fee.
Assess is always first
Always first. What is being logged is never what anyone remembers it being, and the volume is rarely what anyone expects.
In your own repositories and infrastructure, staged, with the existing system kept running until the new one is trusted.
Dashboards, alert rules, runbooks, and your team able to add a rule without us.
What we do, and who else does it
| Option | Stack chosen for your estate | Cost does not scale with what you log | Every alert has a runbook | You keep it when it ends | It actually gets finished | Best fit when… |
|---|---|---|---|---|---|---|
| A hosted SaaS platform | you want it running this week and your volume is predictable. | |||||
| Your own team does it | the work is genuinely prioritised, and stays prioritised. | |||||
| A generalist contractor | you already know what you want built and can specify it. | |||||
| Doing nothing | no incident, audit or renewal has asked for it yet. | |||||
| OpsHero | you want it sized, alerting on something real, and handed back. |
Not every estate needs a stack of its own
A small single-service estate is genuinely well served by the cloud provider’s own tooling, and we would rather tell you that than sell you a stack.
Tick the ones that are true
Four quick questions. Your answers stay in your browser.
Starting with the one everybody opens with
Which stack would you pick for us?
It depends — and here is exactly on what, so that answer is not evasion. How much you log. Whether full-text search matters or label lookup is enough. What retention you are actually obliged to keep. How high your metric cardinality runs. And what your team can operate at 3am. Give us those five and the recommendation is usually straightforward.
Can you reduce our observability bill?
Usually yes. The levers are retention tiering, sampling, cardinality control, and sometimes moving to a different engine. We will not quote you a percentage — the honest version of that number comes from your estate, and you will have it after the assessment.
Elastic or Loki?
Elastic where search matters — full-text query across structured logs, with access control people can live with. Loki where the estate is Kubernetes-shaped and label lookup gets you to the right stream, at much lower storage cost. If most of your queries start with a service name and a time range rather than a phrase, that is Loki.
Why Fluent Bit and not Logstash?
Footprint. It runs on every node without a memory argument, which is why it displaced Logstash in Kubernetes estates.
Do we have to leave our current SaaS?
No. Some clients keep it and shrink what they send to it — often the cheapest win available, and it requires no migration at all.
Will you run it for us?
No. These are scoped engineering engagements and we are not your on-call. We build it, hand it over with runbooks, and your team operates it.
Can you fix our alert noise without losing coverage?
Yes, and it is one of the most common engagements we take. Rules get rewritten against what actually happened in your incidents rather than what someone feared might.
Talk to an engineer
Not a salesperson. You will get a read on what is being collected, what it costs, and what would change.
- What is actually being logged, and at what volume
- What it costs today, and which levers apply to that bill
- A stack recommendation for your estate, with the reasoning
- An honest answer if the provider’s own tooling is enough