Complete Grafana Guide · Part 1 / 4

Prometheus and
Grafana Basics

Collect the numbers, ask questions, draw them
— one lap around a minimal metrics setup

Metrics PromQL Grafana Dashboard docker-compose

CPU usage last 30m

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

Greet the room and note this is part 1 of 4. Say up front that the panel on the right is what they will build themselves by the end of today.

problemwhy monitoring

By the time you type `top`, it's too late

You hear the service got slow and ssh into the box
You can see what the CPU is doing right now
There is no way to know what it looked like 30 minutes ago
Finding the cause needs past numbers, and they were never kept anywhere

$ top
Processes: 612 total, 3 running
CPU usage: 78.4% user, 9.1% sys

# so what about 30 minutes ago?
…no record

What monitoring does Keep piling up the numbers so you can rewind them later.

A slide that builds rapport through shared experience. Asking “who here has run top after an incident?” for a show of hands works well. The point is history, not the live value.

backgroundobservability

Observability splits into three kinds of data

Pillar	What it is	Typical tools	Today
Metrics	Numbers over time — CPU, memory, request counts	Prometheus, Datadog	covered
Logs	Event records — error messages, request logs	Loki, ELK Stack	not covered
Traces	Request paths — calls between services	Tempo, Jaeger	Part 3

Today's scope Metrics only. Collecting numbers with Prometheus and drawing them with Grafana.

Do not try to explain all three. Draw the line — “logs and traces belong to other parts” — and move on. Just remember logs come back in the cardinality slide (15).

series4 parts

Where today sits in the series

Part	Title	Area
Part 1	Prometheus and Grafana Basics	Metrics basics
Part 2	Custom Metrics in a Go Application	Metrics in depth
Part 3	Distributed Tracing with Grafana Tempo	Traces
Part 4	Continuous Profiling with Grafana Pyroscope	Profiles

Some screens today borrow metrics from the sample app built in part 2. The environment we set up today shows node-exporter and Prometheus' own metrics.

Saying up front that the Counter/Histogram/Summary screenshots come from the Go sample app in part 2 prevents “why can't I see this?” questions later.

contentsagenda

What we cover today

01 — Prometheus core concepts

Pull-based architecture
The four metric types
Labels, time series, cardinality

02 — PromQL

The four pieces of a query
instant vector vs range vector
Three real queries, dissected

03 — Grafana

Connecting a Data Source
Dashboard · Row · Panel · Query
Reusing dashboards with Variables

04 — Hands-on setup

One docker-compose file to run it all
Scraping node-exporter metrics
A four-panel system dashboard

Chapters 1 and 2 are concepts, 3 and 4 are hands-on. If time runs short, trim the real-world queries (24–26) and protect chapter 4.

01

Prometheus core concepts

How metrics come in, and what shape they are stored in. Get this and PromQL and dashboards are just stories told on top of it.

This chapter is the root of the talk. Do not rush the definition of “one time series” (14) and cardinality (15) — they are what prevent accidents later.

01 · Prometheusarchitecture

It fetches metrics itself — the Pull model

As long as a target exposes a /metrics HTTP endpoint, Prometheus calls that address on a schedule (scrape) and pulls the values. The target never has to know where to send anything.

Node Exporter:9100/metrics

Go Application:8080/metrics

MySQL Exporter:9104/metrics

scrapeevery 15s

Prometheus:9090 · TSDB · PromQL engine

query

Grafana:3000 · dashboards

Alertmanager:9093 · alert rules

Slack / Emailnotify

The arrows show which way the data flows. The calls go the other way — Prometheus calls each target's /metrics.

Calling an exporter a “translator” lands well: it turns a system's state into text Prometheus can read. Today's lab runs only node-exporter.

01 · Prometheuspull vs push

What Pull and Push trade away

Aspect	Pull — Prometheus	Push — Datadog, InfluxDB
Data flow	Server fetches from targets	Targets send to the server
Service discovery	Needed — must know what to scrape	Not needed — targets send directly
Network	Server → target reachability	Target → server reachability
Upside	Target health for free — failed scrape = down	Works behind a firewall
Downside	Hard to scrape targets behind firewalls	Missing data from a dead target is hard to spot

One line to remember With Push, when a target goes quiet you cannot tell an outage from no traffic. With Pull, the failed call is itself the signal.

Directly tied to quiz Q1. You can also mention the exception: short-lived workloads such as Kubernetes CronJobs suit Pull poorly, and that is what Pushgateway is for.

01 · Prometheusmetric types

There are only four metric types

Type	Behavior	Example use	Key functions
Counter	Only goes up (back to 0 on restart)	HTTP request count, error count	rate(), increase()
Gauge	Goes up and down	CPU usage, memory, active connections	read directly, avg_over_time()
Histogram	Spreads values across buckets	Response time, request size	histogram_quantile()
Summary	Stores pre-computed quantiles	Response time (client-side)	read directly

In practice You will not use all four. The two you actually reach for are Counter and Histogram.

Just skim the table here. Announce that the next five slides show each type on a real graph.

01 · metric typescounter

Counter — the answer is the slope, not the value

http_requests_total

All you can read is “it keeps going up”. The drop to the floor at 14:11 is an application restart.

rate(http_requests_total[5m])

5–10 per second — the real traffic shows. The line stays smooth across the restart — counter resets are corrected automatically.

Rule See a Counter, think rate() first.

Grafana panel plotting the raw cumulative http_requests_total

Grafana panel showing the per-second rate computed with rate()

Click both buttons to compare two faces of the same data. Asking the audience “can you read the current request rate off the cumulative graph?” before switching to rate() lands hard. Ties to quiz Q2.

01 · metric typesgauge

Gauge — the value itself is the answer

It is the value at this instant, so it goes up and down
CPU usage, memory used, active connections
Unlike a Counter, there is no reason to wrap it in rate()

# in-flight requests right now
http_requests_in_flight

# 5-minute average
avg_over_time(node_cpu_seconds_total{mode="idle"}[5m])

Here too you can see memory dip at the 14:11 restart and fill back up.

The most intuitive type, so move fast. But slide 23 has the trap that rate() on a Gauge raises no error, so drive “no reason to use it” home here.

01 · metric typeshistogram

Histogram — buckets are cumulative

Stores only counts per bucket; percentiles are computed at query time
le="0.5" means everything under 0.5s, not the 0.25–0.5s slice
For the distribution, set the panel format to Heatmap

# P99 response time
histogram_quantile(0.99,
  rate(http_request_duration_seconds_bucket[5m]))

How to read it Dark bands are where requests pile up: 10–25ms (read API) and 250–500ms (order-create). An average would smear both into one number near 100ms.

Heatmap panel showing the response time distribution

The “trap of averages” slide. A bimodal distribution smeared into one mean resonates everywhere. Ties to quiz Q3.

01 · metric typessummary

Summary — already computed, so it cannot be merged

Aspect	Histogram	Summary
Quantile computed	Server (at query time)	Client (at collection)
Merge across instances	Possible	Impossible
Change buckets/quantiles	Edit config, restart	Code change required
Recommended for	Most cases	When exact quantiles are a must

Rule of thumb Averaging the p99 of three servers does not give you a p99. With more than one server, use a Histogram.

Summary panel plotting go_gc_duration_seconds per quantile label

GC pause times the Go runtime exports by default. The median sits near 100µs while the max swings past 500µs.

It is fine to sum Summary up as “you will rarely use it”. Still, Go runtime metrics expose one, so they should recognize it. Ties to quiz Q4.

01 · Prometheustime series

The unit of storage is one time series

A time series = one name tag with (timestamp, value) pairs appended forever. Every dot on a graph is one row.

time       value
14:30:00   1,204
14:30:15   1,251     ← one row every 15s (scrape interval)
14:30:30   1,298
14:30:45   1,344

One name tag only tells you “requests went up” — not which API, or how many failed. So we attach labels.

http_requests_total{method="GET" , path="/api/orders", status="200"}
http_requests_total{method="POST", path="/api/orders", status="201"}
http_requests_total{method="POST", path="/api/orders", status="500"}

The point Same name, but Prometheus stores these as three separate time series. Labels are what make splitting and merging possible.

Prometheus Table tab showing one metric split out by label combination

One metric name, but as many time series as there are label combinations.

This is the most important concept in the talk. Using PromQL while believing “metric name = time series” guarantees getting stuck on aggregation (22) and division (25).

01 · Prometheuscardinality

The shadow of labels — series grow by multiplication

method5×path20×status6

0 time series — that much is fine

+ user_id100,000and you get

0

Every single series costs memory. Prometheus simply falls over. This is called a cardinality explosion.

One test decides it Can you count this label's possible values in advance?

Fine as a label

method
status
path

Send it to logs

user_id
session_id
email · IP address

Pause while the number climbs so the audience reads the digits. Stress that this is the number one cause of dead Prometheus servers in the field. Ties to quiz Q5.

01 · Prometheusnaming

Where do those names come from?

http_requests_total is not built into Prometheus. Every one of these names was typed into someone's code.

var HttpRequestsTotal = prometheus.NewCounterVec(
    prometheus.CounterOpts{
        Name: "http_requests_total",
        Help: "Total number of HTTP requests",
    },
    []string{"method", "path", "status"},
)

The third argument lists the label names this metric will use. To Prometheus a name is just a string, so my_stuff_counter would have worked identically.

Official naming rules	Example
One domain word up front	node_ · process_ · prometheus_
Plural unit at the end	_seconds · _bytes
Cumulative counts end in _total	http_requests_total

Where a name comes from	Who picks it
node_cpu_seconds_total	exporter author
go_gc_duration_seconds	registered by the library
http_requests_total	application developer

No need to memorize names. Ask with curl -s :9090/api/v1/metadata, or use the autocomplete in the Prometheus UI.

“Nobody declared them, so why do go_ metrics exist?” comes up a lot. The answer: attaching the Go client library brings the runtime metrics along with it.

01 · Prometheusexposition format

What Prometheus actually scrapes

# TYPE http_requests_total counter
http_requests_total{method="GET",path="/api/orders",status="200"} 272
http_requests_total{method="POST",path="/api/orders",status="201"} 64
http_requests_total{method="POST",path="/api/orders",status="500"} 8

# TYPE http_requests_in_flight gauge
http_requests_in_flight 0

# TYPE http_request_duration_seconds histogram
..._bucket{le="0.1"}   7
..._bucket{le="0.25"}  32
..._bucket{le="0.5"}   72
..._bucket{le="+Inf"}  72
..._sum   19.577
..._count 72

# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0.5"} 4.82e-05
go_gc_duration_seconds_sum   8.74e-05
go_gc_duration_seconds_count 2

The # TYPE line is where the type comes from. It starts with # but it is metadata, not a comment
A Counter gets one line per label combination. Three lines here, so three series
A Histogram fans one name out over many lines. Buckets climbing 7 → 32 → 72 and then holding is what cumulative means — all 72 finished within 0.5s
A Summary has no buckets. Only finished numbers, so there is no raw material to merge
A Gauge is one current value and that is it

Showing the raw text after all the concepts pulls the whole story together at once. If the lab is up, curling it live here is the best option.

02

PromQL

A Grafana panel is, in the end, one line of PromQL. There is no building a dashboard without it.

Open with “it looks like a cipher, but the structure is simple”. Breaking it into four pieces on the next slide is the spine of this chapter.

02 · PromQLanatomy

Every query is four pieces put together

sum by(path) (rate(http_requests_total{status="500"}[5m]))

metric namewhat to look at

label filterwhich series of those

time rangeover what window

function · aggregationhow to compute them

Read it from the inside out Out of http_requests_total → keep only status="500" → take the last 5 minutes → turn it into a per-second rate → sum it by path.

Use this slide as the anchor for the chapter. Saying that the following slides each zoom into one piece keeps the flow clear.

02 · PromQLselector

Pieces 1 · 2 — the conditions inside the braces

Operator	Meaning	Example
=	value is equal	{method="GET"}
!=	value differs	{device!="lo"}
=~	matches the regex	{status=~"5.."}
!~	does not match it	{path!~"/health.*"}

A common idiom In {status=~"5.."}, . is any single character. 500, 502 and 503 all match — that is, “all 5xx”.

# comma-separated conditions must all match
http_requests_total{method="GET", status=~"5.."}

Dropping loopback with device!="lo" on a network panel, or picking one filesystem with mountpoint="/" on a disk panel — it all happens right here.

Ties to quiz Q7. Adding that regexes are fully anchored rather than partial matches makes it precise.

02 · PromQLinstant vs range

Piece 3 — what changes when `[5m]` is attached

The most confusing spot in PromQL, because every function demands a different shape of input.

instant vector — one value, right now

http_requests_total
# 1 value per series (e.g. 1,344)

Enough when you just want to look at a Gauge.

range vector — every value in the window

http_requests_total[5m]
# 20 values per series (15s × 5m)

rate() is “increase ÷ elapsed time”, so it needs at least two values.

rate(http_requests_total[5m])   ✓ works
rate(http_requests_total)       ✗ error

How wide should the window be It needs at least two samples, so keep it comfortably longer than the scrape interval. At 15s, a minute or more; 5m by convention. Too short and it twitches, too long and change blurs out.

Ties to quiz Q6. “Why is [5m] there?” always comes up. Settle it here before moving on.

02 · PromQLaggregation

Piece 4 — putting the split series back together

Function	Meaning
sum()	adds them all up
avg()	takes the average
max() / min()	largest / smallest
count()	counts how many series

sum(rate(http_requests_total[5m]))
→ total traffic · 1 line

sum by(path) (rate(http_requests_total[5m]))
→ traffic per path · one line per path

sum by(path, status) (...)
→ per path × status combination

by and without by = keep only these labels. without = drop these labels and keep the rest.

Ties to quiz Q8. That a bare sum(...) throws away every label and collapses to a single value foreshadows the error-rate slide (25).

02 · PromQLfunctions · trap

Four functions cover most of it — and one trap

Function	What it does	Metric type	Example
rate()	per-second average rate	Counter	rate(http_requests_total[5m])
increase()	total increase over a window	Counter	increase(http_requests_total[1h])
histogram_quantile()	computes a percentile	Histogram	histogram_quantile(0.99, rate(...))
avg_over_time()	average over a window	Gauge	avg_over_time(metric[5m])

The trap — that “metric type” column is never checked The Prometheus server does not know whether a metric is a Counter or a Gauge. It flattens every type into untyped floating-point time series.

rate(node_memory_MemAvailable_bytes[5m])
→ no error. Computes on a Gauge (result is nonsense)

rate(http_requests_total)
→ error. not a range vector

Errors catch the vector type only; matching the metric type is on you. The name is a decent guess — _total means Counter, a _bucket/_sum/_count set means Histogram, a quantile label means Summary.

This is the classic path to a quietly wrong dashboard. Worth adding that rate() on a Gauge reads every dip as a counter reset and “corrects” it away.

02 · query 1cpu %

CPU usage — one layer at a time from the inside

100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

1

node_cpu_seconds_total{mode="idle"}

Out of cumulative CPU time, pick only the idle mode

2

[5m] + rate(...)

Slope over 5 minutes = idle seconds per second. With 8 cores you get one series per core

3

avg by(instance)

Average the per-core values per host. Result is 0–1; 1 means completely idle

4

* 100

Turn it into a percentage

5

100 - (...)

Flip “percent idle” into “percent busy”

Running the CPU usage query in the Prometheus Graph tab

It is faster to check in the Prometheus UI that a query returns what you want before building the dashboard.

The point is that no metric measures CPU usage directly, so we invert idle. It looks odd at first, but explaining that Linux reports it that way settles it.

02 · query 2error rate · trap

Error rate — without `sum()` you always get 1

sum(rate(http_requests_total{status=~"5.."}[5m]))
  / sum(rate(http_requests_total[5m])) * 100

Why sum() is needed Division in PromQL pairs series whose label sets match exactly. Without sum(), the left {status="500"} pairs with the right {status="500"} — dividing something by itself, so always 1.

# without sum — always 1
{method="POST", path="/api/orders", status="500"}  1

# after sum strips the labels — the real rate
{}  5.5

The point is removing status from the labels, so instead of sum() you can name what to keep with by. For a per-path error rate:

sum by(path) (rate(http_requests_total{status=~"5.."}[5m]))
  / sum by(path) (rate(http_requests_total[5m])) * 100

Generalized Before dividing two queries, ask first — do both sides carry the same label set?

This is 90% of the “my error rate is stuck at 100%” reports. Tell them a graph pinned flat at 1.0 should raise this suspicion.

02 · query 3memory · disk

Memory and disk — half the job is picking the metric

(1 - node_memory_MemAvailable_bytes
     / node_memory_MemTotal_bytes) * 100

No rate() — both are Gauges, so the current value is the answer
Both sides carry the same instance, job labels, so they pair up by themselves. That is where this parts ways with the error-rate query
MemAvailable, not MemFree — Linux spends spare memory on file cache, and that cache is reclaimed the moment it is needed. Compute with MemFree and a well-cached server always looks out of memory

(1 - node_filesystem_avail_bytes{mountpoint="/"}
     / node_filesystem_size_bytes{mountpoint="/"}) * 100

Same shape of calculation as memory
Filesystem metrics get a separate series per mount point, so pin one with mountpoint. Skip it and you get /, /boot and Docker overlays too
avail, not free — Linux reserves some blocks for root only. What an ordinary process can really use is avail

Lab note The node-exporter started by docker-compose is boxed in its container and sees only its own filesystem. If the disk query is empty, check the real mount points with curl -s :9100/metrics | grep "^node_filesystem_size_bytes".

MemAvailable vs MemFree trips up even long-time Linux users, so it gets a good reaction. Mentioning that Docker Desktop on macOS/Windows cannot see the host disk cuts questions during the lab.

03

Grafana

An open-source dashboard tool that plugs into many data sources and visualizes them. Besides Prometheus it can talk to Loki, Tempo, MySQL and Elasticsearch.

Point out that Grafana stores nothing. Prometheus stores, Grafana draws.

03 · Grafanadata source

Data Source — register it first

Option 1 — from the UI

Connections → Data sources → Add Pick Prometheus URL: http://prometheus:9090 Save & Test

The point prometheus:9090, not localhost:9090 — Grafana runs in a container too, so it must reach it by container name.

# Option 2 — auto-register via provisioning
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true

The Prometheus Data Source settings screen in Grafana

Save & Test → “Successfully queried the Prometheus API”.

Ties to quiz Q9. If anyone in the room is new to container networking, add the line: on a docker-compose network the service name is the hostname.

03 · Grafanadashboard · panel

Dashboard → Row → Panel → Query

Dashboard can be exported/imported as JSON ├─ Row collapsible group of panels │ ├─ Panel one unit of visualization │ │ └─ Query one line of PromQL │ └─ Panel └─ Row

One line to remember One panel = one PromQL query. Building a dashboard is really writing queries.

Panel type	Good for
Time series	CPU usage, request rate, response time
Stat	Current uptime, total requests
Gauge	Disk usage, memory usage
Table	Per-instance status, Top N lists
Bar chart	Errors per service, requests per endpoint
Heatmap	Response time distribution, Histogram buckets

Exporting a dashboard as JSON matters a lot in practice — mention managing them as code and shipping them via provisioning.

03 · Grafanavariables

Variables — one dashboard for twenty servers

Field	Value
Name	instance
Type	Query
Data source	Prometheus
Query	label_values(node_cpu_seconds_total, instance)
Multi-value	enabled

100 - (avg by(instance) (rate(node_cpu_seconds_total{
    mode="idle", instance=~"$instance"}[5m])) * 100)

Pick an instance from the dropdown and only that instance's metrics remain. Add servers and the list grows with whatever label_values() finds.

The instance variable dropdown at the top of the dashboard

With one server nobody feels the need, but by the time you are cloning a dashboard per server it is already too late.

04

Hands-on — a local setup

Bring up Prometheus · Grafana · node-exporter with a single docker-compose file, then build a four-panel system monitoring dashboard.

Live demo from here if at all possible. Pull the images beforehand, and be ready to fall back on the screenshots in the following slides.

04 · hands-ondocker-compose.yml

Three services and you are done

services:
  prometheus:
    image: prom/prometheus:v2.51.0
    ports: ["9090:9090"]
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.retention.time=7d'

  grafana:
    image: grafana/grafana:11.4.0
    ports: ["3000:3000"]
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin
    volumes:
      - ./provisioning:/etc/grafana/provisioning

  node-exporter:
    image: prom/node-exporter:v1.8.1
    ports: ["9100:9100"]

> docker compose up -d

Service	Address
Prometheus	localhost:9090
Grafana	localhost:3000 · admin/admin
node-exporter	localhost:9100/metrics

Retention --storage.tsdb.retention.time=7d — 7 days for a local lab. In production this is traded against disk.

Point out that the image tags are pinned. Leave them on latest and yesterday's working lab breaks today.

04 · hands-onprometheus.yml

`scrape_configs` decides what gets scraped

global:
  scrape_interval: 15s      # scrape period
  evaluation_interval: 15s  # rule evaluation period

scrape_configs:
  # Prometheus' own metrics
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # node-exporter system metrics
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['node-exporter:9100']

scrape_interval: 15s — this becomes the spacing between points in a time series
job_name becomes the job label attached to every series
This uses static_configs, but real operations wire in service discovery such as Kubernetes or Consul

The price of Pull The server has to know “where to scrape” — that is this file. Push setups have no such file.

Connecting “service discovery needed” from the Pull vs Push table (8) to this concrete file is what makes it click.

04 · hands-onstatus → targets

When something breaks, look here first

Prometheus web UI → Status → Targets
Endpoint, last scrape time and how long it took are all here
If this says DOWN, no amount of dashboard tweaking gives you anything but empty graphs

Debugging order ① Are the targets UP → ② does the query return values in the Prometheus UI → ③ only then the Grafana panel settings. Skip the order and you burn time.

Two targets showing UP on the Prometheus Targets page

Ties to quiz Q10. “My graph is empty” is the most common question during the lab, and the answer is almost always on this screen.

04 · hands-onfirst dashboard

The minimum setup for watching one server

# 1. CPU usage (%)  · Unit: Percent (0-100)
100 - (avg by(instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# 2. Memory usage (%)
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

# 3. Disk I/O  · Unit: Bytes/sec (IEC)
rate(node_disk_read_bytes_total[5m])
rate(node_disk_written_bytes_total[5m])

# 4. Network traffic  · Unit: Bytes/sec (IEC)
rate(node_network_receive_bytes_total{device!="lo"}[5m])
rate(node_network_transmit_bytes_total{device!="lo"}[5m])

If the legend looks messy Disk and network produce one series per device. Keep only real devices with device=~"nvme.*|sd.*", or turn the legend off.

System monitoring dashboard with four panels: CPU, memory, disk and network

Stress that skipping Unit leaves the y-axis as bare numbers that are hard to read. Setting a Unit per panel changes dashboard quality dramatically.

wrap-uprecap

What to take away today

Prometheus is Pull-based — it scrapes the /metrics endpoints exporters expose
There are four metric types, but the ones you actually reach for are Counter and Histogram
PromQL is mostly covered by four functions: rate() · increase() · histogram_quantile() · avg_over_time()
Labels slice a metric into dimensions, but a unique identifier blows cardinality up
A Grafana dashboard is Data Source → Dashboard → Row → Panel → Query
Prometheus + Grafana + node-exporter come up from a single docker-compose file

Reading the six lines aloud is enough of a recap on its own. It is fine to start taking questions here.

checkclick to reveal

Answer these yourself

The application restarted and the counter began again at 0. rate() corrects for that break, so the line does not jump there. → Slide 10

Every request of 0.5s or less, not just the 0.25–0.5s slice. Buckets are cumulative, so values keep growing to the right. → Slide 12

Histogram. A Summary only hands over quantiles each server already computed, so they cannot be merged — averaging three p99s is not the overall p99. → Slide 13

Series count grows as the product of label value counts. 600 series times 100,000 users is 60 million. Values you cannot count belong in logs. → Slide 15

Without a range, each series yields one current value (an instant vector). “Increase ÷ elapsed time” needs at least two, so add [5m] to make it a range vector. → Slide 21

Status → Targets in the Prometheus web UI. If a target is DOWN there is no data at all, so touching queries or panel settings is pointless. → Slide 34

Let the audience answer first, then click to open. If time is short, Q4 and Q6 alone are enough.

nextpart 2

Next — Custom Metrics in a Go Application

One step past the system metrics node-exporter gives you, we instrument a Go application directly. HTTP request counts and response times, of course, plus business metrics like order volume, all the way up into Grafana.

Among today's screens The Counter · Histogram · Summary screenshots came from that very sample app.

References

Full code — github.com/kenshin579/tutorials-go /monitoring/grafana-metrics
Prometheus docs — prometheus.io/docs
PromQL querying guide — prometheus.io/docs/prometheus/latest/querying/basics
Grafana docs — grafana.com/docs/grafana/latest
node-exporter — github.com/prometheus/node_exporter
Dashboard Best Practices — grafana.com/docs/grafana/latest/dashboards/build-dashboards/best-practices

Thank you Questions welcome — advenoh.pe.kr

Take Q&A with the GitHub repo URL up on the screen.

Prometheus andGrafana Basics

By the time you type top, it's too late

Observability splits into three kinds of data

Where today sits in the series

What we cover today

Prometheus core concepts

It fetches metrics itself — the Pull model

What Pull and Push trade away

There are only four metric types

Counter — the answer is the slope, not the value

Gauge — the value itself is the answer

Histogram — buckets are cumulative

Summary — already computed, so it cannot be merged

The unit of storage is one time series

The shadow of labels — series grow by multiplication

Where do those names come from?

What Prometheus actually scrapes

PromQL

Every query is four pieces put together

Pieces 1 · 2 — the conditions inside the braces

Piece 3 — what changes when [5m] is attached

Piece 4 — putting the split series back together

Four functions cover most of it — and one trap

CPU usage — one layer at a time from the inside

Error rate — without sum() you always get 1

Memory and disk — half the job is picking the metric

Grafana

Data Source — register it first

Dashboard → Row → Panel → Query

Variables — one dashboard for twenty servers

Hands-on — a local setup

Three services and you are done

scrape_configs decides what gets scraped

When something breaks, look here first

The minimum setup for watching one server

What to take away today

Answer these yourself

Next — Custom Metrics in a Go Application

Shortcuts

Prometheus and
Grafana Basics

By the time you type `top`, it's too late

Piece 3 — what changes when `[5m]` is attached

Error rate — without `sum()` you always get 1

`scrape_configs` decides what gets scraped