🎓 Lesson 15
D5
Reinforcement Learning for Adaptive Process Optimization
Reinforcement learning is a type of AI that learns the best way to control a mining process—like blasting or conveyor speed—by trying actions, seeing what works, and improving over time, just like learning to ride a bike.
🎯 Learning Objectives
- ✓ Explain how reward shaping influences convergence in blast optimization RL agents
- ✓ Design a simplified Q-learning agent architecture for burden adjustment in a digital twin of a surface mine
- ✓ Analyze convergence behavior of an RL policy using episode reward curves and regret metrics
- ✓ Apply state-space discretization techniques to map blast parameters (burden, spacing, powder factor) into RL observation vectors
- ✓ Evaluate trade-offs between exploration and exploitation when deploying RL controllers in safety-critical blasting operations
📖 Why This Matters
In modern mines, blast performance varies daily due to rock mass heterogeneity, weather, and equipment wear—making static designs suboptimal. Traditional optimization methods (e.g., RQD-based empirical formulas) cannot adapt in real time. Reinforcement learning powers the next generation of autonomous blasting systems: digital twins that learn from each blast round, adjust drill-and-blast parameters before the next round, and continuously improve fragmentation uniformity while reducing flyrock and oversize. This isn’t sci-fi—it’s deployed at Rio Tinto’s Gudai-Darri mine and BHP’s South Flank operation.
📘 Core Principles
RL centers on four elements: (1) Agent—the controller (e.g., blast optimizer module), (2) Environment—the simulated or physical mine system (e.g., digital twin of bench geometry + rock properties), (3) State—a vector capturing current conditions (e.g., [RQD, UCS, moisture %, previous fragmentation index]), and (4) Action space—discrete or continuous decisions (e.g., 'increase burden by 0.2 m' or 'select charge type C'). The agent learns a policy π(a|s) mapping states to actions to maximize expected discounted reward Σγᵗrₜ. Key algorithms include Q-learning (for discrete actions) and Deep Deterministic Policy Gradient (DDPG) for continuous control. Crucially, RL differs from supervised learning: it learns from delayed, sparse rewards (e.g., ‘+5 pts’ for passing fragmentation sieve test at crusher) rather than labeled data.
📐 Q-Learning Update Rule
The fundamental equation driving tabular RL learning: it updates the estimated value of taking action a in state s based on observed reward and the best future value. Used to train agents that recommend burden/spacing adjustments after each blast round in simulation.
Q-Learning Bellman Update
Q(s,a) ← Q(s,a) + α [ r + γ · maxₐ' Q(s',a') − Q(s,a) ]Updates the action-value function estimate for state s and action a using observed reward r, discount factor γ, and best estimated future value.
Variables:
| Symbol | Name | Unit | Description |
|---|---|---|---|
| Q(s,a) | Action-value function | dimensionless score | Expected cumulative discounted reward for taking action a in state s and following optimal policy thereafter |
| α | Learning rate | unitless (0 < α ≤ 1) | Controls step size during Q-value updates; higher values adapt faster but risk instability |
| γ | Discount factor | unitless (0 ≤ γ < 1) | Weights immediate vs. future rewards; γ = 0.9–0.99 typical for mining control tasks |
| r | Immediate reward | points or normalized score | Scalar feedback received after executing action a in state s |
Typical Ranges:
Blast optimization digital twin: α = 0.05 – 0.15
Long-horizon scheduling RL: γ = 0.95 – 0.99
💡 Worked Example
Problem: A digital twin simulates blast rounds. At state s = [RQD=65%, UCS=82 MPa], agent chooses action a = 'reduce burden by 0.15 m'. It receives immediate reward r = +3.2 (based on predicted P80 improvement). Next state s' = [RQD=67%, UCS=80 MPa]. Current Q(s,a) = 4.1. Learning rate α = 0.1, discount γ = 0.95. Max Q(s',a') = 4.8.
1.
Step 1: Identify known values — Q(s,a) = 4.1, r = 3.2, α = 0.1, γ = 0.95, max Q(s',a') = 4.8
2.
Step 2: Apply Q-learning update: Qₙₑw(s,a) = Q(s,a) + α[r + γ·max Q(s',a') − Q(s,a)]
3.
Step 3: Compute: Qₙₑw = 4.1 + 0.1[3.2 + 0.95×4.8 − 4.1] = 4.1 + 0.1[3.2 + 4.56 − 4.1] = 4.1 + 0.1×3.66 = 4.1 + 0.366 = 4.466
Answer:
The updated Q-value is 4.466, reflecting improved expectation of long-term reward for this state-action pair. This value falls within the typical convergence range of 3.0–6.5 for well-tuned blast optimization agents.
🏗️ Real-World Application
At Newmont’s Boddington Mine (Western Australia), an RL-powered digital twin was integrated with blasthole survey data, real-time geotechnical logs, and post-blast LiDAR fragmentation scans. The RL agent—trained offline on 12 months of historical blasts—recommends burden and spacing adjustments per drill pattern zone. After deployment, oversize (>75 cm) reduced by 22%, crusher throughput increased 9%, and rework drilling decreased 17%. Critically, the agent used a safety-constrained reward function: penalties applied for predicted flyrock energy > 12 kJ/m² or stemming violation risk > 5% (per SME guidelines).
✏️ Student Exercise
You are tasked with designing the reward function for an RL agent optimizing explosive powder factor (kg/t) in a copper porphyry open pit. Given constraints: target P80 ≤ 35 cm (measured via post-blast photogrammetry), maximum allowable ground vibration = 12 mm/s (PPV), and minimum fragmentation uniformity index (FUI) ≥ 0.72. Define a composite reward r = w₁·f₁(P80) + w₂·f₂(PPV) + w₃·f₃(FUI), where each fᵢ is normalized [0,1] and weights reflect priority (safety > fragmentation > cost). Calculate r for a trial blast yielding P80 = 38 cm, PPV = 10.4 mm/s, FUI = 0.69. Use linear penalty scaling: f₁ = max(0, 1 − (P80−35)/15), f₂ = max(0, 1 − (PPV−12)/12), f₃ = max(0, 1 − (0.72−FUI)/0.1), with w₁=0.4, w₂=0.45, w₃=0.15.