The SUMO traffic
Simulation of Urban MObility (SUMO) is an open-source library that allows modeling of intermodal traffic systems including road vehicles, public transport, and pedestrians.
In this blog, I tell how I used SUMO’s powers to attempt a solution for a very common problem:
‘This traffic light is red for 60 seconds. I have to wait even when there are no vehicles on the other roads”
or
“Why is there a traffic jam on this signal at this time of the day, every day?. Can’t they see there is so much traffic on this route. They should reduce signal wait time for this peak time.”
Yes, this problem can be solved using reinforcement learning. I used the Double Deep Queued Network (DDQN) algorithm to obtain a near-perfect simulation of traffic on a network of roads where waiting time at traffic lights is determined by the reinforcement learning agent. The implementation of RL logic is beyond the scope of this blog.
SUMO helps in building our solution by providing a suitable medium to create traffic simulation.
SUMO comes packed with the following features.
The OSMWeb Wizard of SUMO is used to generate the map. I imported the OpenStreetMap of a section of Gwalior city to the wizard and it created this map for me.
Pretty accurate, right? This simulation map has a lot of roads, not a good environment I can tightly control. So I trimmed down the map to have 4 roads and 4 traffic signals.
It supports the simulation of multimodal traffic: cars, trucks, buses, motorcycles, and bicycles. We can control the appearance of each type of vehicle by altering properties in XML files generated by the OSMWeb Wizard. It also supports adding traffic lights.
Cars are yellow, bikes are green and all other heavy motor vehicles are red. As we can see from this image, if a vehicle is stopped, the vehicle’s tail lights become red. The broad white bar indicates traffic lights. When the bar is white it means the signal is red, if it is grey the signal is green.
This was all about creating the simulation.
How does my Reinforcement Learning agent know this simulation state?
Using SUMO, I create the initial state of the simulation environment and run it. I need to connect my RL Agent to this simulation. I do this using TraCI. You can fetch the required properties of the simulation from the Traci object using getters.
For example, I use the count of each type of vehicle to define my agent’s state. This is how I run my agent.
import tracisumoBinary = “/path/to/sumo-gui”
sumoCmd = [sumoBinary, “-c”, sumoConfig, “ — start”]traci.start(sumoCmd)trafic_light_ids = traci.trafficlights.getIDList()
actions_map = make_map(trafic_light_ids)detector_ids = traci.inductionloop.getIDList()
state_space_size = traci.inductionloop.getIDCount() * 2
action_space_size = len(actions_map)agent = Learner(state_space_size, action_space_size, 0.0)state = get_state(detector_ids)total_reward = 0
step = 0
while step < 1000:
action = agent.act(state)
lights_phase = actions_map[action] for light, index in zip(trafic_light_ids, range(len(trafic_light_ids))):
traci.trafficlights.setPhase(light, lights_phase[index]) for i in range(2):
traci.simulationStep() next_state = get_state(detector_ids)
reward = calculate_reward(state, next_state)
total_reward += reward state = next_state step+=2traci.close()
For every step, the agent generates the next state of the simulation by following one of the many actions (turn off the red light, turn off the green light, etc.) it can perform and calculates reward. The agent learns the best next action from the calculated reward. The learning policy is defined by the DDQN algorithm.
After every step, the GUI simulation is also changed as per the state chosen by the agent and we view what our agent is learning by the flow of simulation states.