Redis Real-Time Decisioning Demo | Oil & Gas Midstream

Stage 1: The Architecture

Real-time pipeline anomaly decisioning stack.

Natural gas pipelines generate continuous telemetry across hundreds of measurement points. When pressure or flow deviates from expected, the control center has seconds to determine whether the event is a normal operational transient or an incident requiring isolation. Redis assembles the live network state, segment history, anomaly signals, and operating context before the operator finishes reading the alarm.

Data Sources

SCADA / OT Historians

Real-time telemetry — pressure, flow rate, temperature, compressor status — from RTUs and flow computers across the pipeline network

Integrity Database

Inspection records, anomaly history, pipe specifications, cathodic protection data, ILI results — Postgres

Compressor / Equipment PLCs

Live equipment state — compressor run status, discharge pressure, speed, vibration, seal condition

GIS / Network Model

Pipeline topology, segment attributes, elevation, valve locations, interconnect points

Kafka / Event Stream

SCADA alarms, nomination changes, equipment state transitions — streaming in real time

Ingest Layer

Redis Data Integration (RDI)

CDC from Postgres integrity database — inspection history, anomaly records, segment specifications — YAML pipelines, sub-second lag, 0 custom ETL

Redis Streams / Kafka

Live SCADA telemetry, alarm events, compressor state changes — streaming into Redis RAM as they occur

Redis Context · Pipeline Operational Store

Redis RAM

Live pipeline state — current pressure/flow readings, active alarms, compressor run status, recent telemetry history

Redis Flex

5-year operating history, maintenance records, ILI inspection results, historical transient library at tiered cost

Redis Search

Anomaly pattern library, similar historical event vectors, segment risk profiles

Pipeline Anomaly Decision Engine

Anomaly Classifier (ML)

Distinguishes incident signatures from operational transients — trained on historical SCADA events

Threshold Rules Engine

MAOP, flow deviation thresholds, rate-of-change limits, multi-sensor correlation rules

Network Hydraulic Context

Current flow path alternatives, compressor available capacity, alternate routing options

Policy Engine

IMP-mandated response thresholds, regulatory classification rules, operator-configured response procedures

Output Channels

Control Center Dashboard

ISOLATE / MONITOR / NORMAL recommendation with full evidence chain

Field Operations

Valve closure sequence, crew dispatch, isolation procedure

Integrity Management System

Anomaly log, regulatory notification trigger, post-event investigation queue

Learn: Confirmed incident outcomes → anomaly classifier retraining → pattern library refresh in Redis Flex

Decision SLA

<10 ms

Pipeline coverage

Every segment monitored

False positive cost

$47K/hr per isolation

Stage 2: Anomaly Detected

Pressure deviation on Segment 7 — incident or transient?

A 180-mile natural gas transmission line detects a pressure anomaly on Segment 7 between Compressor Station 4 and Delivery Point 11. Three inline measurement points show corroborating pressure drop. No scheduled valve operations are active upstream. The control center has seconds to determine whether this is a demand-driven transient or a leak requiring immediate isolation.

Anomaly Event

SEG-007 (CS4 → DP-11)

Natural gas transmission · 180 miles · 36-inch diameter · MAOP 1,480 psi · API 5L X70 steel · Last ILI: 14 months ago

ANOMALY DETECTED

Eventpipeline_pressure_deviation

NetworkNorth Texas Transmission — Zone 3

Detected at14:17 local time (3 corroborating sensors)

Segment pressure drop3.4% in 6 minutes

Flow rate variance-8.2% from setpoint

Rate of pressure change-0.34 psi/min (threshold: -0.20)

Upstream valve operationsNone active in last 30 min

Demand signalNo nomination changes filed

Why Real-Time Matters Here

MANUAL CORRELATION

When an anomaly alarm fires, operators must manually correlate SCADA readings, GIS network state, integrity database records, compressor logs, and nomination data across 5+ separate systems. This typically takes 8–12 minutes. During that time, the pipeline continues to operate — and if the event is a real incident, the gap between detection and isolation is where gas escapes and regulatory exposure accumulates.

REDIS DECISION ENGINE

Full segment context — 5 years of operating history, inspection records, live pressure trend, and the anomaly pattern library — assembled in 9.2ms. The decision arrives before the operator has finished reading the alarm banner.

Stage 3: Ingest

Segment telemetry, inspection records, and network state flow into Redis.

RDI continuously syncs Segment 7's inspection history, pipe specifications, and historical anomaly records from the integrity database — YAML pipelines, sub-second CDC lag, zero custom sync code. Redis Streams and Kafka bring in live SCADA alarms and pressure readings. The last ILI inspection result from 14 months ago is already in Redis Flex when this alarm fires.

Redis Data Integration (RDI)Redis Streams / Kafka

Source Systems → Redis (Zone 3 / SEG-007)

SCADA / OT Historian — real-time telemetry + alarms

Pressure, flow rate, temperature across 3 inline sensors on Segment 7 — streaming via Redis Streams/Kafka

→

Postgres / Integrity DB — inspection records + anomaly history

ILI results, corrosion measurements, anomaly classifications, repair history, MAOP documentation

→

PLC

Compressor PLCs — equipment state

CS4 compressor run status, discharge pressure, speed, seal condition — event-driven via RDI

→

GIS

GIS / Network Model — pipeline topology

Segment geometry, valve locations, elevation profile, interconnect nodes, alternate routing options

→

KFK

Redis Streams / Kafka — live SCADA events

Alarm notifications, pressure deviation events, equipment state transitions

→

RDI Pipeline Status

CDC modeReal-time (Debezium)

Integrity DB lagunder 1 sec

SCADA event lagunder 300 ms

Segment coverageAll 23 segments, Zone 3

Deployment modelOn-premises (OT-isolated network)

Custom sync code0 lines (YAML only)

Why this matters for pipeline operations: IMP-mandated response times require rapid classification of anomalous conditions. The difference between an incident response and a false positive investigation is the quality of context available at decision time. RDI ensures inspection history, prior anomaly records, and equipment state are always current in Redis — no batch refresh, no manual data pull.

Stage 4: Context

Segment 7's full operating context assembles around this anomaly.

Redis RAM holds the live pipeline state — current pressure readings across all three sensors, active alarm history, upstream compressor status. Redis Flex surfaces five years of operating history and the last ILI results. Redis Search scans the historical transient library for event signatures that match the current pressure/flow pattern. All three stores respond before the operator's second screen refreshes.

Redis RAMRedis FlexRedis Search

Segment Profile (Pipeline 360)

Segment age22 years · Last ILI: 14 months ago

Pipe specifications36-inch, X70 steel, wall thickness 0.432 in

MAOP1,480 psi · Current operating pressure: 1,241 psi (84%)

Last anomaly (historical)Minor metal loss indication, 26 months ago — monitored

Prior false positives3 in last 18 months (all demand-driven transients)

ILI result summaryNo significant defects — B31G remaining life >30 years

Live Anomaly Signals

Pressure drop rate-0.34 psi/min (3 sensors corroborating)

Flow variance-8.2% below setpoint

Upstream valve activityNone — no operational explanation

Nomination changesNone filed — no demand explanation

CS4 compressorNormal — discharge pressure and speed stable

Anomaly pattern match (Search)0.87 similarity — 2021 Segment 12 leak event

Context signal: Three corroborating sensors, no operational explanation (no valve operations, no demand changes, no compressor anomaly), and a 0.87 pattern match to a confirmed 2021 incident on Segment 12 — a similar pressure/flow signature that was classified as a leak. The absence of an operational explanation is itself the most significant signal. None of this was connected before Redis.

Stage 5: Feature Serving

6 anomaly classification features assembled in 4.3ms.

Redis RAM serves the live pressure telemetry and alarm state. Redis Flex surfaces the historical incident record and inspection baseline. Redis Search scores the current event signature against the anomaly pattern library in under a millisecond. The ML classifier sees the same feature definitions used during model training — zero drift between training and production inference.

Redis SearchRedis Flex

pressure_rate_change_psi_min

Rate of pressure decline across 3 inline sensors — psi per minute

-0.34 psi/min (threshold: -0.20)0.3 ms

flow_deviation_3sensor_corr

Flow rate deviation from setpoint with 3-sensor correlation confirmation

-8.2% (corroborated — 3/3 sensors)0.2 ms

upstream_valve_activity_30m

Count of upstream valve operations in prior 30 minutes

0 (no operational explanation)0.2 ms

demand_nomination_delta

Change in gas nominations filed affecting this segment

0 (no demand explanation)0.3 ms

anomaly_pattern_similarity

Cosine similarity to confirmed incident signatures in Redis Search library

0.87 — SEG-012 2021 leak0.3 ms

segment_inspection_risk_score

Composite segment risk score from last ILI results and age

0.52 (moderate — 14-month-old ILI)0.4 ms

TOTAL FEATURES

P99 LATENCY

4.3 ms

ANOMALY LIBRARY

847 patterns

CLASSIFIER ACCURACY

94.2%

Stage 6: Decision

Three response paths scored. ISOLATE SEG-007 recommended in 6.1ms.

The anomaly classifier evaluates three possible responses — isolate and investigate, monitor with elevated alert, or classify as normal transient and clear. The pattern library match at 0.87, combined with zero operational explanation and 3-sensor corroboration, drives the ISOLATE recommendation. The policy engine confirms the IMP response threshold is exceeded.

Anomaly ClassifierPolicy Engine

6 anomaly features evaluated

847-pattern library queried

0 operational explanations found

Decision returned in 6.1ms

#1 Recommendation

ISOLATE SEG-007

Isolate Segment 7 · Reroute via C-12 → C-14

Confidence 0.87 — incident signature confirmed

3-sensor corroboration, no operational explanation, 0.87 pattern match to confirmed 2021 incident. IMP response threshold exceeded. Recommended: close block valves BV-47 and BV-52, reroute flow via alternate compression path CS4 → C-12 → C-14 → DP-11. Alternate path has 340 MMcf/d available capacity.

Confidence87%

#2 Elevated Monitor

MONITOR

Elevated alert — continue operating with enhanced monitoring

Applied when pattern match is 0.50–0.75

Would apply if operational explanation partially accounted for the deviation. Current pattern match at 0.87 exceeds the monitor threshold. No operational explanation reduces confidence in a transient classification.

Path score13%

#3 Suppressed

NORMAL TRANSIENT

Clear alarm — classify as operational transient

Suppressed — 3-sensor corroboration with no operational explanation

Policy engine suppresses normal-transient classification when 3 or more sensors corroborate AND upstream valve activity is zero AND nomination changes are zero. All three suppression criteria are met.

Path score0%

Stage 7: Business Impact

9.2ms between anomaly detection and isolation decision.

Without real-time context assembly, operators spend 8–12 minutes manually correlating SCADA, GIS, integrity database, compressor logs, and nomination data before they can classify an anomaly. At a 73% false positive rate across the network, that delay is mostly spent on transients — but for the 27% that are real incidents, every minute of delay means more gas escaped, more regulatory exposure, and a longer post-incident investigation.

Anomaly Response Timeline

MANUAL CORRELATION PATH (WITHOUT REDIS)

14:17Alarm fires — operator sees pressure deviation

14:17 → 14:29Manual correlation across SCADA, GIS, integrity DB, compressor logs, nominations

14:29Incident classification completed (12 minutes later)

14:29Isolation order issued

Outcome12-minute gap between detection and isolation · Gas continued to escape · Regulatory clock started at 14:17, not 14:29

REDIS DECISION ENGINE PATH

14:17Anomaly detected

14:17 + 9.2msFull segment context assembled — ISOLATE recommendation issued

14:17 + 10msIsolation order displayed to operator with full evidence chain

14:17 + 90sOperator confirms — block valves BV-47 and BV-52 close

Outcome9.2ms to decision · Full evidence chain already assembled · 11+ minutes removed from response path

The false positive problem: At a 73% false positive rate, every unnecessary isolation costs approximately $47,000/hour in lost throughput. But the cost of a false negative — a real incident misclassified as a transient — is orders of magnitude higher: regulatory fines, remediation, and reputational consequences. Redis gives operators the context to distinguish the two in milliseconds, not minutes.

Per-Incident Impact

12 min

to isolation decision · manual correlation across 5+ systems

→

9.2ms

ISOLATE recommended · full evidence chain assembled

OPERATOR WORKLOAD

Context assembly eliminated from the decision path — operators validate recommendations, they don't build the case from scratch

REGULATORY RESPONSE

Full audit trail assembled at decision time — IMP documentation starts at 14:17 + 9.2ms, not at the end of manual investigation

Stage 8: Outcome

Same anomaly. Two response paths.

The manual correlation path takes 12 minutes to assemble the context needed to classify the anomaly — by then the pipeline has been leaking for 12 minutes with no isolation in place. The Redis path delivers the full evidence chain in 9.2ms, enabling an immediate isolation recommendation the operator can act on within 90 seconds.

Manual Correlation (Without Redis)

CONTROL CENTER

ANOMALY ALARM — CONTEXT ASSEMBLY IN PROGRESS

Manual correlation in progress

Context assembly12+ min (5 manual system queries)

ILI inspection accessManual DB query (4–6 min)

Historical pattern matchNot performed (no tooling)

Isolation decision12 min after anomaly

Gas released during gapUnconstrained

OUTCOME

DELAYED

Isolation issued 12 minutes after detection · Operator manually assembled context from 5 systems · Regulatory exposure: 12-minute detection-to-response gap

12 min

to decision

5 systems

queried manually

−$94K

est. 2hr throughput loss

Redis Decision Engine

CONTROL CENTER

ISOLATE RECOMMENDED — 9.2ms

9.2ms

SEG-007 — North Texas Transmission Zone 3

Incident Pattern Confirmed · Isolation Recommended

3-sensor corroboration. No operational explanation. Pattern match 0.87 vs 2021 Segment 12 incident. ISOLATE SEG-007 — reroute via C-12 → C-14.

Context assembly9.2 ms (full segment 360)

ILI inspection access0.4 ms (Redis Flex)

Historical pattern match0.87 — SEG-012 2021 (Redis Search)

Isolation recommendation9.2 ms after anomaly

Alternate pathCS4 → C-12 → C-14 → DP-11 ready

OUTCOME

ISOLATED

Decision in 9.2ms · Full evidence chain assembled before operator reads alarm · Block valves BV-47 and BV-52 closed at 14:17 + 90s

9.2ms

to decision

0 manual

queries required

Incident contained

evidence chain complete

Stage 9: Architecture Recap

Every segment. Every anomaly. Sub-10ms context assembly.

The architecture is designed to repeat across every segment in the transmission network. RDI keeps inspection records and operating history always fresh. Redis RAM holds the live network state. Redis Flex holds the historical incident library and ILI results at tiered cost. Redis Search scans the anomaly pattern library in under a millisecond. The decision engine always sees the full context — and the operator gets a recommendation before they finish reading the alarm.

Data Sources

SCADA / OT Historians

Real-time telemetry — pressure, flow rate, temperature, compressor status — from RTUs and flow computers across the pipeline network

Integrity Database

Inspection records, anomaly history, pipe specifications, cathodic protection data, ILI results — Postgres

Compressor / Equipment PLCs

Live equipment state — compressor run status, discharge pressure, speed, vibration, seal condition

GIS / Network Model

Pipeline topology, segment attributes, elevation, valve locations, interconnect points

Kafka / Event Stream

SCADA alarms, nomination changes, equipment state transitions — streaming in real time

Ingest Layer

Redis Data Integration (RDI)

CDC from Postgres integrity database — inspection history, anomaly records, segment specifications — YAML pipelines, sub-second lag, 0 custom ETL

Redis Streams / Kafka

Live SCADA telemetry, alarm events, compressor state changes — streaming into Redis RAM as they occur

Redis Context · Pipeline Operational Store

Redis RAM

Live pipeline state — current pressure/flow readings, active alarms, compressor run status, recent telemetry history

Redis Flex

5-year operating history, maintenance records, ILI inspection results, historical transient library at tiered cost

Redis Search

Anomaly pattern library, similar historical event vectors, segment risk profiles

Pipeline Anomaly Decision Engine

Anomaly Classifier (ML)

Distinguishes incident signatures from operational transients — trained on historical SCADA events

Threshold Rules Engine

MAOP, flow deviation thresholds, rate-of-change limits, multi-sensor correlation rules

Network Hydraulic Context

Current flow path alternatives, compressor available capacity, alternate routing options

Policy Engine

IMP-mandated response thresholds, regulatory classification rules, operator-configured response procedures

Output Channels

Control Center Dashboard

ISOLATE / MONITOR / NORMAL recommendation with full evidence chain

Field Operations

Valve closure sequence, crew dispatch, isolation procedure

Integrity Management System

Anomaly log, regulatory notification trigger, post-event investigation queue

Learn: Confirmed incident outcomes → anomaly classifier retraining → pattern library refresh in Redis Flex

Decision SLA

<10 ms

Pipeline coverage

Every segment monitored

False positive cost

$47K/hr per isolation