HeavyKeeper hot-key remediation

HeavyKeeper detection feeds hot-key remediation. Remediation policies are conservative by default: the built-in default is observe-only, and side-effecting action mode requires explicit operator enablement.

Policy scopes and precedence

Scope	Example	Precedence
Table	`TABLE app.events`	Highest
Namespace / keyspace / schema / database	`KEYSPACE app`, `SCHEMA app`, `DATABASE app`, `NAMESPACE app`	Middle
Cluster	`CLUSTER`	Lowest explicit
Built-in default	synthetic observe-only policy	Fallback

Policy fields

Field	Meaning
`mode`	`observe_only` or `action`.
`threshold`	Remediation skew threshold; distinct from detector alert threshold.
`window_ms`	Evidence window for confidence.
`min_confidence`	Required confidence for action planning.
`cooldown_ms`	Cooldown before another action.
`action_ttl_ms`	Runtime action lifetime.
`allowed_actions`	Duplicate-free supported action families.
`max_active_remediations`	Planning-time cap.
`rate_limit_max_inflight`	Required when `rate_limit` is allowed.
`rate_limit_retry_after_ms`	Required when `rate_limit` is allowed.

Set and unset policies

SET HOT_KEY_REMEDIATION CLUSTER
WITH (
  mode = 'observe_only',
  threshold = 1000,
  window_ms = 60000,
  min_confidence = 0.8,
  cooldown_ms = 300000,
  action_ttl_ms = 300000,
  allowed_actions = [],
  max_active_remediations = 0
);

SET HOT_KEY_REMEDIATION KEYSPACE app
WITH (
  mode = 'action',
  threshold = 5000,
  window_ms = 60000,
  min_confidence = 0.95,
  cooldown_ms = 300000,
  action_ttl_ms = 300000,
  allowed_actions = ['rate_limit'],
  max_active_remediations = 8,
  rate_limit_max_inflight = 32,
  rate_limit_retry_after_ms = 50
);

SET HOT_KEY_REMEDIATION TABLE app.events
WITH (
  mode = 'observe_only',
  threshold = 2500,
  window_ms = 120000,
  min_confidence = 0.9,
  cooldown_ms = 600000,
  action_ttl_ms = 300000,
  allowed_actions = [],
  max_active_remediations = 0
);

Equivalent scoped ALTER form:

ALTER TABLE app.events
SET HOT_KEY_REMEDIATION
WITH (
  mode = 'observe_only',
  threshold = 2500,
  window_ms = 120000,
  min_confidence = 0.9
);

Unset restores inheritance from the next lower-precedence policy or the built-in default:

UNSET HOT_KEY_REMEDIATION TABLE app.events;
UNSET HOT_KEY_REMEDIATION KEYSPACE app;
UNSET HOT_KEY_REMEDIATION CLUSTER;

Show, explain, and expire

SHOW HOT_KEY_REMEDIATION FOR TABLE app.events;

EXPLAIN HOT_KEY_REMEDIATION FOR TABLE app.events;

EXPIRE HOT_KEY_REMEDIATION ACTION 'action-id';

Operational rules

Rule	User-facing effect
Observe-only allows candidates but no routing side effects.	You can inspect hot keys safely before action mode.
Policy identity is pinned to candidates/actions.	Historical generations remain auditable.
Table policy overrides namespace policy, which overrides cluster policy.	Local hot-key behavior can be scoped narrowly.
Dropped observations block action planning until a complete clean evidence window exists.	Backpressure does not accidentally trigger unsafe action.
Raw hot-key bytes are not metric/log labels.	Observability uses bounded fingerprints and scoped counters.
Strict CQL compatibility rejects remediation extensions.	Enable MTS extensions to use this syntax.