AI & ML / UCT · CSC3022 / 2024
Learning to navigate Four Rooms
Explore state, reward and sequential decisions through tabular Q-learning agents collecting packages across deterministic and stochastic rooms.
UCT / COURSE CONTEXT / 2024
CSC3022F · C++ and Machine Learning
C++ language and memory-model foundations followed by Python-based machine-learning implementation.
2026 PORTFOLIO RECREATIONS
Four RoomsENGINEERING PERSPECTIVE
Inside the work
Explore sequential decision-making: a move changes both position and collection progress, and an immediate reward must be weighed against later rewards. Three scenarios make state design and termination visible.
- NumPy / Python
- Store and update tabular state-action values in submitted scenario agents.
- Supplied FourRooms environment
- Grid transitions, package placement and path output; environment authorship is not claimed.
Encode the situation
Represent the grid position and scenario-specific collection state in the table index.
- Bellman-style update
- Combine immediate reward with a discounted maximum next-state value to update the chosen Q entry.
- State sufficiency
- Coordinates and remaining/order-dependent package status must distinguish states with different future objectives.
- Exploration
- The report mentions epsilon-greedy, but the retained agents choose argmax. Random environment transitions are not an epsilon-greedy policy.
What the demonstration establishes
Recorded playback shows positions and actual package events. Manual browser movement explains the environment, not live training or proof of convergence; bounded recovery runs are separate from historical plots.
Evidence behind this page
Three scenario sources, named report and six retained path plots. Code takes precedence over unsupported report claims.
What it is
Three submitted agents explore single, multiple and ordered package collection in a supplied Four Rooms environment.
Why it was built
Explore how state-action values can drive package collection in deterministic and stochastic room transitions.
My contribution
Submitted scenario agents and Q-table/reward logic; the FourRooms environment is supplied teaching code. Environment redistribution is held pending permission.
How it works
Use NumPy state-action tables, reward shaping and Q-value updates. Actions are greedy argmax selections; stochastic environment transitions provide a separate source of randomness.
- 01Grid state
- 02Greedy action
- 03Transition / reward
- 04Q-value update
- 05Path plot
Technical implementation
- NumPy stores state-action values. Scenario variants change package-count/order objectives and reward handling.
- The agent chooses greedy argmax actions; stochastic transitions in the environment are distinct from epsilon-greedy exploration, which is not claimed here.
- The Q update combines immediate reward with discounted next-state values. State bookkeeping and terminal updates need care when package status changes.
- A post-project wrapper bounds runtime and writes regenerated paths separately from the retained six original plots.
Inside the agent’s path
0 / 15 · Recorded completed episode
Fresh 2026 execution of original Scenario1 after its training loop. Greedy actions; stochastic environment may substitute another action with 20% probability. Default mode uses genuine stored transitions, not live browser training. Rewards: blocked −15, package +80, other move −1. One retained run per mode is not a success-rate evaluation.
Source update equation. Individual Q values were not retained and are not fabricated.
Visual evidence
Results & evidence
- Six original deterministic/stochastic path images are retained.
- A bounded launcher isolates fresh generated paths from historical evidence.
Engineering challenges
A coordinate alone may not capture collection progress. The original loops, state stride and terminal-update behaviour constrain what the path evidence can establish.
Scope & limitations
Bookkeeping, terminal updates and unbounded loops can affect completion. A successful single run is not a convergence or optimality result. No invented success-rate metric is included.
What I learned
The practical importance of state design, exploration, reward shaping and termination in reinforcement learning.
What I would improve now
Review Markov state encoding, add explicit exploration and compare seeded completion rates across scenarios. Do not infer convergence from a successful path.