Real-Time Decisioning Demo | Oil & Gas Downstream — Refinery Process Optimization
Pipeline<10ms
1 / 9
1
Architecture
2
Crude Change
3
Ingest
4
Context
5
Features
6
Decision
7
Impact
8
Outcome
9
Recap
Stage 1: The Architecture
Real-time refinery process optimization stack.
Refineries already have sophisticated control, historian and optimization systems. The Redis opportunity is not to replace them — it is to ensure the optimization models always see current process conditions, current economics, and current crude slate before they recommend a setpoint change. Redis sits between the data sources and the optimizer, staging the context a decision needs so the optimizer never runs on stale data.
Data Sources

DCS / Process Historians

Real-time process telemetry — temperatures, pressures, flow rates, compositions, column draws — from distributed control systems across all refinery units

LIMS (Laboratory)

Crude assay results, product quality measurements, off-spec notifications, blend component specs — Postgres

Crude Assay Database

Crude slate composition, API gravity, sulfur content, distillation curves, feed nominations — Postgres

Databricks / ML Platform

Offline-trained yield models, APC setpoint models, energy optimization models — lakehouse + feature materialization

SAP / ERP

Feedstock economics, product prices, hedged volumes, utility costs, planning targets

Ingest Layer

Redis Data Integration (RDI)

CDC from Postgres LIMS, crude assay DB, and ERP — crude quality, economics, product specs — YAML pipelines, sub-second lag, 0 custom ETL

Redis Streams / Kafka

Real-time DCS telemetry events, feed quality change notifications, LIMS result arrivals

Redis Context · Refinery Optimization Store

Redis RAM

Live process conditions — current CDU temperatures, live crude blend composition, active unit setpoints, real-time feed rate

Redis Flex

2-year crude/yield history, APC model output archive, economics baseline, equipment performance history at tiered cost

Redis Search

Crude run pattern library, similar operating condition vectors, yield deviation signatures

Refinery Optimization Engine

Yield / Margin Model

Economic yield optimization across crude cuts — trained on historical run data, served online via Redis

APC Setpoint Model

Advanced Process Control recommended operating targets for CDU, vacuum unit, and secondary units

Constraint Engine

Equipment limits, product spec requirements, safety envelope, utility capacity

LP Optimizer Interface

Translates model outputs into APC-compatible setpoint recommendations

Output Channels

APC / DCS

Updated operating targets — CDU overflash temperature, side-draw rates, column pressure setpoints

Operator Advisory

Recommendation rationale, yield impact, acceptance interface

Planning System

Yield update, actual vs. plan variance, economics refresh

Learn: Actual yield vs. predicted → yield model retraining → optimization baseline refresh in Redis Flex
Decision SLA
<10 ms
Optimization coverage
All major process units
Model refresh lag
Sub-second (RDI)
Stage 2: Crude Change
Crude slate shifting to Arab Light — yield opportunity window is open.
A Gulf Coast refinery CDU is receiving a changing crude blend over the next four hours as Arab Light shipments arrive and Arab Medium inventory draws down. LIMS just confirmed the feed API gravity is rising. The APC system is still running on targets optimized for the previous slate. The optimization engine has a window to recalculate setpoints — but only if it can see the current crude quality and current economics right now.
Crude Slate Change Event
CDU
CDU-1 (Crude Distillation Unit 1)
Gulf Coast Refinery · 110,000 BPD capacity · Running at 87% · Current slate: Arab Medium/Light blend
OPTIMIZATION TRIGGER
Eventcrude_slate_change_detected
RefineryGulf Coast Refinery — CDU-1
Detected at09:41 AM (LIMS assay result received)
Current API gravity33.4° (↑2.8° in 4 hours — trending Arab Light)
Sulfur content1.87% wt (↓0.41% — favorable shift)
Current overflash target362°F (optimized for prior slate)
Estimated margin opportunity$31,200/day incremental yield
APC model data ageT-4 hours (batch features — stale)
Why Real-Time Matters Here
BATCH OPTIMIZATION
Refinery optimization models typically run on historian data that is 4–8 hours old. When the crude slate changes, the optimizer doesn't know until the next batch run. The CDU continues on targets that are correct for the previous crude — naphtha yield is left on the table and margin accumulates on the wrong side of the decision.
REDIS + OPTIMIZER
Current LIMS assay, live CDU operating conditions, and yield model features assembled in 7.4ms. The optimizer sees the current crude before the APC system has cycled again — and issues updated targets while the opportunity window is still open.
Stage 3: Ingest
Crude assay, process conditions, and economics flow into Redis.
RDI continuously syncs crude assay results from LIMS and feedstock economics from SAP — YAML pipelines, sub-second CDC lag, zero custom ETL. Redis Streams brings in live DCS telemetry and LIMS result notifications. The new Arab Light assay that just arrived from the lab is already in Redis when the optimization trigger fires.
Redis Data Integration (RDI)Redis Streams / Kafka
Source Systems → Redis (CDU-1)
DCS
DCS / Process Historian — real-time process telemetry
CDU temperature profiles, pressure, flow rates, draw rates, reflux
→
LI
LIMS — crude assay + product quality
Arab Light/Medium assay results, API gravity, sulfur, distillation curves
→
CA
Crude Assay DB — feedstock specifications
Crude slate composition, API gravity history, distillation curve library — Postgres CDC via RDI
→
ML
Databricks — yield model features
Offline-trained APC and yield optimization models materialized to Redis online feature store
→
ERP
SAP / ERP — feedstock economics
Arab Light/Medium pricing, product values, utility costs — CDC via RDI
→
RDI Pipeline Status
CDC modeReal-time (Debezium)
LIMS CDC lagunder 1 sec
Crude assay DB lagunder 1 sec
ERP economics lagunder 2 sec
Deployment modelOn-premises (refinery network / OT edge)
Custom sync code0 lines (YAML only)
Why this matters for refinery optimization: Crude assay results from the lab arrive in Redis before the current APC cycle has completed. The optimizer is never deciding on yesterday's crude — it sees this morning's blend the moment LIMS posts the result.
Stage 4: Context
CDU-1's full operating context assembles around the crude change.
Redis RAM holds the live process conditions — current CDU temperatures, live crude blend composition, active APC setpoints. Redis Flex surfaces two years of crude run history and the yield model archive. Redis Search finds similar crude run patterns where the yield opportunity was successfully captured.
Redis RAMRedis FlexRedis Search
CDU Profile (Optimization 360)
Unit capacity110,000 BPD · Current rate: 95,700 BPD (87%)
Current crude blendArab Medium 61% / Arab Light 39% (shifting)
Current overflash target362°F (last updated 4 hours ago)
Naphtha side-draw4.2% of feed (current)
Last APC cycle4.2 hours ago (stale for current slate)
Similar run historyArab Light dominant runs available in Redis Search
Live Process Signals
LIMS API gravity (latest)33.4° — Arab Light blend increasing
Sulfur (latest)1.87% wt — favorable for light cuts
CDU feed rate95,700 BPD — stable, no constraint
Overflash current vs. optimal362°F vs. 357°F (Arab Light target)
Estimated naphtha giveaway~0.8% light cut below Arab Light optimal
Similar run match (Search)0.89 similarity — Arab Light dominant 2024-Q3 run
Context signal: Arab Light composition shift has moved the optimal overflash target 5°F below the current APC setpoint. The difference represents ~0.8% naphtha yield gap — $31,200/day at current prices. Redis Search confirms a 2024-Q3 run at near-identical blend where the overflash adjustment captured the full opportunity. None of this context was connected before Redis.
Stage 5: Feature Serving
89 yield model features hydrated in 3.9ms.
Redis Flex holds the yield model feature archive and crude run history. Redis Search finds the matching historical run pattern. Redis RAM serves the live process conditions and current economics.
Redis SearchRedis Flex
crude_api_gravity
Current feed API gravity from LIMS — primary driver of light cut potential
33.4° (Arab Light dominant)0.3 ms
crude_sulfur_pct
Feed sulfur content from LIMS — impacts downstream unit loading
1.87% wt (↓ favorable)0.2 ms
overflash_temp_delta
Difference between current APC target and model-optimal for current crude
+5°F above optimal (362 vs. 357)0.2 ms
light_cut_yield_deviation
Actual vs. model-predicted light cut yield for current conditions
-0.8% below Arab Light optimal0.3 ms
similar_run_yield_uplift
Achieved yield improvement in closest historical similar crude run (Redis Search)
+0.82% light cut in 2024-Q3 run0.4 ms
margin_delta_per_bbl
Incremental margin per barrel from overflash target adjustment at current product prices
+$0.33/bbl at current naphtha price0.3 ms
TOTAL FEATURES
89
P99 LATENCY
3.9 ms
MODEL REFRESH
Sub-second (RDI)
TRAIN/SERVE PARITY
100%
Stage 6: Decision
Two operating paths scored. Overflash target reduction recommended in 5.7ms.
The yield optimizer evaluates two operating paths — adjust CDU targets for the current Arab Light blend, or maintain current APC setpoints. It scores each path against expected light cut yield, margin impact, and downstream unit constraints.
Yield ModelConstraint Engine
89 yield model features evaluated
Current slate confirmed: Arab Light dominant
Constraint check: all limits clear
Decision returned in 5.7ms
#1 Recommendation
ADJUST CDU TARGETS
Overflash 362°F → 357°F · Naphtha draw +0.4%
Yield improvement: +0.8% light cut · $31,200/day
Overflash target reduction releases lighter fractions that are currently leaving in the bottom draw. At Arab Light composition (33.4° API), the 357°F overflash target captures the additional naphtha yield. Downstream constraint check passed — no reformer or hydrotreater capacity issue at the higher naphtha draw rate.
Optimization score0.91
#2 Maintain Targets
HOLD SETPOINTS
Hold current APC targets — 362°F overflash
Applied when crude shift is within blend tolerance
Maintaining current targets is appropriate when the crude API shift is within ±1°. Current shift of +2.8° is above the blend tolerance threshold — maintaining targets means continuing to leave naphtha yield on the table while the Arab Light composition persists. Scores 0.09.
Optimization score0.09
#3 Evaluated
FULL REOPTIMIZATION
Trigger full LP reoptimization cycle
Triggered when crude change exceeds 5° API shift
Full LP reoptimization warranted for major crude slate changes (>5° API or >1% sulfur shift). Current shift below that threshold — targeted APC adjustment is the appropriate response.
Optimization score0%
Stage 7: Business Impact
7.4ms between crude change detection and optimization recommendation.
Without real-time context, the optimizer runs on 4-hour-old features. Crude slate changes take a full batch cycle to reach the model. With Redis, the LIMS assay lands in the optimizer's feature vector the moment it arrives from the lab.
Optimization Timeline
BATCH OPTIMIZATION PATH
09:41 AMLIMS confirms Arab Light shift — queued
09:41 → 13:47CDU runs 362°F for 4+ hours on Arab Medium targets
13:47 PMBatch feature export runs — LIMS data finally included
14:02 PMOptimizer recalculates — 4h21m after crude change
Outcome$5,575 missed margin ($31,200/day × 4.3/24)
REDIS + OPTIMIZER PATH
09:41 AMLIMS result arrives
09:41 AM + under 1sRDI syncs assay to Redis — optimizer context updated
09:41 AM + 7.4ms89 features assembled — 357°F recommendation to APC
09:53 AMCDU on Arab Light optimal targets
Outcome$31,200/day yield improvement captured
The platform play: CDU-1 is one unit in a refinery with dozens of optimization opportunities — vacuum unit, FCC, reformer, hydrotreater, alkylation. Redis is the context layer that ensures every optimizer in the refinery always sees current process conditions. The compound effect across all units is what changes the economics conversation.
Per-Slate-Change Impact
$5,575
missed margin · batch path · 4.3 hours of suboptimal targets
→
$31,200/day
yield improvement captured · 7.4ms to decision
CONTINUOUS OPTIMIZATION
Every crude blend change, every LIMS result, every economics update triggers an optimization check — not just the scheduled batch cycle
DOWNSTREAM RIPPLE
Overflash optimization cascades to downstream units — lighter crude means vacuum unit and FCC feed quality also improves when the CDU is on the right targets
Stage 8: Outcome
Same crude change. Two optimization paths.
The batch path means the CDU runs on Arab Medium targets for 4+ hours after the crude shifts. The Redis path delivers the updated recommendation 7.4ms after the LIMS assay arrives.
Batch Optimization (Without Redis)
OPTIMIZATION RUNNING — T-4H FEATURES
STALE FEATURES ACTIVE
Crude assay data 4 hours old at decision time
Crude assay data ageT-4h (stale)
LIMS lag4+ hours (batch export)
Overflash target362°F — Arab Medium optimized
Naphtha yield−0.8% (suboptimal for Arab Light)
Opportunity captured$0 (window missed)
OUTCOME
SUBOPTIMAL
CDU running Arab Medium targets on Arab Light crude · 4.3 hours of suboptimal yield · $5,575 in incremental margin missed
$5,575
margin missed
4.3 hr
suboptimal run
−0.8% yield
naphtha giveaway
Redis + Optimizer
REDIS + OPTIMIZER
SETPOINT UPDATED — 7.4ms
7.4ms
CDU-1 — Gulf Coast Refinery
Arab Light Blend Confirmed · Targets Adjusted
LIMS assay confirmed Arab Light shift. Overflash 362°F → 357°F. Naphtha draw +0.4%. APC setpoint update issued. Yield improvement: +0.8% light cut.
Crude assay data ageSub-second (RDI)
LIMS lagUnder 1 sec
Overflash target357°F — Arab Light optimized
Naphtha yield+0.8% improvement (Arab Light optimal)
Daily margin impact+$31,200/day
OUTCOME
OPTIMIZED
Arab Light targets active within 12 minutes of crude change · $31,200/day
$31,200/day
yield captured
7.4ms
to setpoint
+0.8% yield
naphtha uplift
Stage 9: Architecture Recap
Every crude change. Every optimization window. Sub-10ms.
The architecture is designed to operate continuously across every major process unit in the refinery. RDI keeps crude assay results, LIMS data, and feedstock economics always fresh — the optimizer never waits for a batch export.
Data Sources

DCS / Process Historians

Real-time process telemetry — temperatures, pressures, flow rates, compositions, column draws — from distributed control systems across all refinery units

LIMS (Laboratory)

Crude assay results, product quality measurements, off-spec notifications, blend component specs — Postgres

Crude Assay Database

Crude slate composition, API gravity, sulfur content, distillation curves, feed nominations — Postgres

Databricks / ML Platform

Offline-trained yield models, APC setpoint models, energy optimization models — lakehouse + feature materialization

SAP / ERP

Feedstock economics, product prices, hedged volumes, utility costs, planning targets

Ingest Layer

Redis Data Integration (RDI)

CDC from Postgres LIMS, crude assay DB, and ERP — crude quality, economics, product specs — YAML pipelines, sub-second lag, 0 custom ETL

Redis Streams / Kafka

Real-time DCS telemetry events, feed quality change notifications, LIMS result arrivals

Redis Context · Refinery Optimization Store

Redis RAM

Live process conditions — current CDU temperatures, live crude blend composition, active unit setpoints, real-time feed rate

Redis Flex

2-year crude/yield history, APC model output archive, economics baseline, equipment performance history at tiered cost

Redis Search

Crude run pattern library, similar operating condition vectors, yield deviation signatures

Refinery Optimization Engine

Yield / Margin Model

Economic yield optimization across crude cuts — trained on historical run data, served online via Redis

APC Setpoint Model

Advanced Process Control recommended operating targets for CDU, vacuum unit, and secondary units

Constraint Engine

Equipment limits, product spec requirements, safety envelope, utility capacity

LP Optimizer Interface

Translates model outputs into APC-compatible setpoint recommendations

Output Channels

APC / DCS

Updated operating targets — CDU overflash temperature, side-draw rates, column pressure setpoints

Operator Advisory

Recommendation rationale, yield impact, acceptance interface

Planning System

Yield update, actual vs. plan variance, economics refresh

Learn: Actual yield vs. predicted → yield model retraining → optimization baseline refresh in Redis Flex
Decision SLA
<10 ms
Optimization coverage
All major process units
Model refresh lag
Sub-second (RDI)