AnyLogic
Expand
Font size
Type Parameters:
ROOT - root model agent type
O - Observation data structure type, see #getObservation(Agent, Object)
A - Action data structure type, see #applyAction(Agent, Object)
C - Configuration data structure type, see #applyConfiguration(Agent, Object)
All Implemented Interfaces:
LearningAgentModelInterface<ROOT>, ReinforcementLearningModel<ROOT,​O,​A,​C>, java.io.Serializable

public abstract class ExperimentReinforcementLearning<ROOT extends Agent,​O,​A,​C>
extends ExperimentCustom
implements ReinforcementLearningModel<ROOT,​O,​A,​C>, LearningAgentModelInterface<ROOT>
Base class for Reinforcement Learning experiments.

This class serves as an interface between the model and the platform (Learning Platform) using Reinforcement Learning to train some Learning Agent which is based on that platform (e.g. artificial neural network). The experiment itself is platform-agnostic and speaks in general terms. Experiment may be used in two forms:
  • exported to the Learning Platform supported by AnyLogic,
  • and used during simulation with an already trained Learning Agent (testing purpose) - together with the Learning Platform-specific connector block from the Reinforcement Learning Library.
The learning process depends on the specific Learning Platform.
Below, we describe an example use: it may perform several simulation runs - the sequence of so called Episodes. The number of Episodes is determined by the Learning Platform. Each Episode contains the following logic:
1. Get the Episode Configuration (used for model initialization) from the Learning Agent (may be absent for models which don't require specific initialization).
2. Create root model agent and initialize it using the Configuration: #applyConfiguration(Agent, Object)
3. Start simulation:
3.1 Check the Stop condition (if true, the Learning Platform will be notified about Episode termination condition)
3.2 Get Observation from the model: #getObservation(Agent, Object)
3.3 Pass the Observation to the Learning Agent and receive the Action (or alternative responses like Stop Episode or Stop learning)
3.4 Apply the Action to the model: #applyAction(Agent, Object)
3.5 Run the model until some Stepping point, which should be defined by a user event or another trigger inside the model. To define this point, call ExperimentReinforcementLearning.takeAction( this );, where this is an existing agent containing the event.
3.6 Repeat to 3.1

If your Learning Platform has finished learning, and you want to test your Learning Agent. To do that, add a block from platform-specific reinforcement learning library to your root (Main) model agent, fill in the properties of that block and ensure there is an event/code inside your model calling the code noted in the p.3.5 above. Then simply run any simulation experiment with your model.

Usage:
  • Define observation, action and configuration data structure classes and corresponding methods (p.3).
  • In order to define the moment when the reinforcement learning should take place, create an event or any other type action with the code described in p. 3.5 above.
Author:
AnyLogic North America, LLC https://anylogic.com
See Also:
Serialized Form

Constructor Summary

Constructors 
ExperimentReinforcementLearning()  
ExperimentReinforcementLearning​(LearningAgentInterface<O,​A,​C> learningAgentInterface)  
Constructor Description

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods 
ROOT createModel()
Is called to obtain a new pre-initialized top-level agent.
abstract ROOT createRoot​(Engine engine)
Is called to obtain a new top-level agent.
java.util.Date date()
Returns the current model date with respect to the start time/date and the model time unit.
void initDefaultRandomNumberGenerator​(Engine engine)
This method should be overridden to initialize random number generator of the given engine
Default implementation set new random generation with random seed - for unique experiments
void run()
static void takeAction​(Agent agent)
This method should be called from the user event associated with the experiment step to be performed.
void takeActionForModel​(ROOT root)
This method is internal and shouldn't be called by user.
it may be removed/renamed in future.
double time()
Returns the current model (logical) time.
double time​(TimeUnits units)
Returns the current model (logical) time.
Modifier and Type Method Description

Methods inherited from class com.anylogic.engine.ExperimentCustom

createEngine, getCommandLineArguments, onError, onError, setCommandLineArguments_xjal, setupEngine_xjal

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface com.anylogic.engine.ReinforcementLearningModel

getDataAccessor

Constructor Detail

ExperimentReinforcementLearning

public ExperimentReinforcementLearning()

ExperimentReinforcementLearning

public ExperimentReinforcementLearning​(LearningAgentInterface<O,​A,​C> learningAgentInterface)

Method Detail

createRoot

@AnyLogicInternalCodegenAPI
public abstract ROOT createRoot​(Engine engine)
Is called to obtain a new top-level agent. This method must be defined in a subclass. Note that the top-level agent should just be constructed in this method, as its create() and start() methods will be called later on.
This method should not be used to define parameters
Parameters:
engine - the simulation engine that will simulate the model

initDefaultRandomNumberGenerator

@AnyLogicInternalCodegenAPI
public void initDefaultRandomNumberGenerator​(Engine engine)
This method should be overridden to initialize random number generator of the given engine
Default implementation set new random generation with random seed - for unique experiments
Parameters:
engine - the engine instance

createModel

@AnyLogicInternalCodegenAPI
public ROOT createModel()
Description copied from interface: ReinforcementLearningModel
Is called to obtain a new pre-initialized top-level agent. This method must be defined in a subclass. Note that the top-level agent should just be constructed in this method, as its create() and start() methods will be called later on.
This method should not be used to define parameters
Specified by:
createModel in interface ReinforcementLearningModel<ROOT extends Agent,​O,​A,​C>

run

public void run()
Description copied from class: ExperimentCustom
Specified by:
run in class ExperimentCustom

takeActionForModel

@AnyLogicInternalAPI
public void takeActionForModel​(ROOT root)
Description copied from interface: LearningAgentModelInterface
This method is internal and shouldn't be called by user.
it may be removed/renamed in future.
Specified by:
takeActionForModel in interface LearningAgentModelInterface<ROOT extends Agent>

takeAction

public static void takeAction​(Agent agent)
This method should be called from the user event associated with the experiment step to be performed. As an argument, pass any existing model agent. For example, when the event is located in the context of some agent, the following code should be used:
ExperimentReinforcementLearning.takeAction( this );.

This function will identify the passed 'agent' argument with the Main model agent ('root'), forcing all RL-related data processing (such as retrieving the observation data) to be done in the context of this agent.
Parameters:
agent - any existing agent

time

public double time()
Returns the current model (logical) time. Time stays constant during an event execution and is only advanced between events.
Returns:
current model time

time

public double time​(TimeUnits units)
Returns the current model (logical) time. Time stays constant during an event execution and is only advanced between events.
Parameters:
units - the time units
Returns:
current model time

date

public java.util.Date date()
Returns the current model date with respect to the start time/date and the model time unit.
Returns:
the current model date