java.lang.Object
- com.anylogic.engine.ExperimentCustom
- - com.anylogic.engine.ExperimentReinforcementLearning<ROOT,O,A,C>

Type Parameters:: ROOT - root model agent type; O - Observation data structure type, see getObservation(Agent, Object); A - Action data structure type, see applyAction(Agent, Object); C - Configuration data structure type, see applyConfiguration(Agent, Object)

All Implemented Interfaces:: LearningAgentModelInterface<ROOT>, ReinforcementLearningModel<ROOT,O,A,C>, java.io.Serializable

public abstract class ExperimentReinforcementLearning<ROOT extends Agent,O,A,C>
extends ExperimentCustom
implements ReinforcementLearningModel<ROOT,O,A,C>, LearningAgentModelInterface<ROOT>

Base class for Reinforcement Learning experiments.

This class serves as an interface between the model and the platform (Learning Platform) using Reinforcement Learning to train some Learning Agent which is based on that platform (e.g. artificial neural network). The experiment itself is platform-agnostic and speaks in general terms. Experiment may be used in two forms:

exported to the Learning Platform supported by AnyLogic,
and used during simulation with an already trained Learning Agent (testing purpose) - together with the Learning Platform-specific connector block from the Reinforcement Learning Library.

The learning process depends on the specific Learning Platform.
Below, we describe an example use: it may perform several simulation runs - the sequence of so called Episodes. The number of Episodes is determined by the Learning Platform. Each Episode contains the following logic:
1. Get the Episode Configuration (used for model initialization) from the Learning Agent (may be absent for models which don't require specific initialization).
2. Create root model agent and initialize it using the Configuration: applyConfiguration(Agent, Object)
3. Start simulation:
3.1 Check the Stop condition (if true, the Learning Platform will be notified about Episode termination condition)
3.2 Get Observation from the model: getObservation(Agent, Object)
3.3 Pass the Observation to the Learning Agent and receive the Action (or alternative responses like Stop Episode or Stop learning)
3.4 Apply the Action to the model: applyAction(Agent, Object)
3.5 Run the model until some Stepping point, which should be defined by a user event or another trigger inside the model. To define this point, call ExperimentReinforcementLearning.takeAction( this );, where this is an existing agent containing the event.
3.6 Repeat to 3.1

If your Learning Platform has finished learning, and you want to test your Learning Agent. To do that, add a block from platform-specific reinforcement learning library to your root (Main) model agent, fill in the properties of that block and ensure there is an event/code inside your model calling the code noted in the p.3.5 above. Then simply run any simulation experiment with your model.

Usage:

Define observation, action and configuration data structure classes and corresponding methods (p.3).
In order to define the moment when the reinforcement learning should take place, create an event or any other type action with the code described in p. 3.5 above.

Author:: AnyLogic North America, LLC https://anylogic.com
See Also:: Serialized Form

Constructor Summary

Constructors
Constructor	Description
`ExperimentReinforcementLearning()`
`ExperimentReinforcementLearning(LearningAgentInterface<O,A,C> learningAgentInterface)`

Method Summary

All Methods Static Methods Instance Methods Abstract Methods Concrete Methods
Modifier and Type	Method	Description
`abstract void`	`applyAction(ROOT root, A action)`	This method must be defined in a subclass to get the data from the given `action` object and apply it to the model (`root`). The method is called each step of reinforcement learning / AI test loop.
`abstract void`	`applyConfiguration(ROOT root, C configuration)`	This method must be defined in a subclass to get the data from the given `configuration` object and apply it as the initial setup to the model (`root`). The method is called only once per the whole model run - at the beginning.
`boolean`	`checkEpisodeStopCondition(ROOT root)`	This method may be defined in a subclass to check the additional stop condition of the Episode.
`abstract A`	`createAction()`	This method must be defined in a subclass - just to create new empty Action object
`abstract C`	`createConfiguration()`	This method must be defined in a subclass - just to create new empty Configuration object
`ROOT`	`createModel()`	Is called to obtain a new pre-initialized top-level agent.
`abstract O`	`createObservation()`	This method must be defined in a subclass - just to create new empty Observation object
`abstract ROOT`	`createRoot(Engine engine)`	Is called to obtain a new top-level agent.
`java.util.Date`	`date()`	Returns the current model date with respect to the start time/date and the model time unit.
`abstract void`	`getObservation(ROOT root, O observation)`	This method must be defined in a subclass to get the data from `root` and write it to the fields of the given `observation`. The method is called each step of reinforcement learning / AI test loop.
`void`	`initDefaultRandomNumberGenerator(Engine engine)`	This method should be overridden to initialize random number generator of the given engine Default implementation set new random generation with random seed - for unique experiments
`void`	`run()`	Use `ExperimentCustom.createEngine()`
`static void`	`takeAction(Agent agent)`	This method should be called from the user event associated with the experiment step to be performed.
`void`	`takeActionForModel(ROOT root)`	This method is internal and shouldn't be called by user. it may be removed/renamed in future.
`double`	`time()`	Returns the current model (logical) time.
`double`	`time(TimeUnits units)`	Returns the current model (logical) time.

Methods inherited from class com.anylogic.engine.ExperimentCustom

createEngine, getCommandLineArguments, onError, onError, setCommandLineArguments_xjal, setupEngine_xjal

Methods inherited from class java.lang.Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

ExperimentReinforcementLearning

public ExperimentReinforcementLearning()

ExperimentReinforcementLearning

public ExperimentReinforcementLearning(LearningAgentInterface<O,A,C> learningAgentInterface)

Method Detail

createRoot

@AnyLogicInternalCodegenAPI
public abstract ROOT createRoot(Engine engine)

Is called to obtain a new top-level agent. This method must be defined in a subclass. Note that the top-level agent should just be constructed in this method, as its create() and start() methods will be called later on.
This method should not be used to define parameters

Parameters:: engine - the simulation engine that will simulate the model

createObservation

@AnyLogicInternalCodegenAPI
public abstract O createObservation()

This method must be defined in a subclass - just to create new empty Observation object

Specified by:: createObservation in interface ReinforcementLearningModel<ROOT extends Agent,O,A,C>
Returns:: new empty Observation object

createAction

@AnyLogicInternalCodegenAPI
public abstract A createAction()

This method must be defined in a subclass - just to create new empty Action object

Specified by:: createAction in interface ReinforcementLearningModel<ROOT extends Agent,O,A,C>
Returns:: new empty Action object

createConfiguration

@AnyLogicInternalCodegenAPI
public abstract C createConfiguration()

This method must be defined in a subclass - just to create new empty Configuration object

Specified by:: createConfiguration in interface ReinforcementLearningModel<ROOT extends Agent,O,A,C>
Returns:: new empty Configuration object

applyConfiguration

@AnyLogicInternalCodegenAPI
public abstract void applyConfiguration(ROOT root,
                                        C configuration)

This method must be defined in a subclass to get the data from the given configuration object and apply it as the initial setup to the model (root).
The method is called only once per the whole model run - at the beginning.

Specified by:: applyConfiguration in interface ReinforcementLearningModel<ROOT extends Agent,O,A,C>
Parameters:: root - the root model agent; action - data structure, coming from the Reinforcement Learning platform and then applied to the model

checkEpisodeStopCondition

@AnyLogicInternalCodegenAPI
public boolean checkEpisodeStopCondition(ROOT root)

This method may be defined in a subclass to check the additional stop condition of the Episode. Should return true to request stopping the Episode (e.g. when the model falls into some undesired terminal state which doesn't allow further training or testing of Learning Agent).

Specified by:: checkEpisodeStopCondition in interface ReinforcementLearningModel<ROOT extends Agent,O,A,C>
Returns:: true to stop the model, false to continue learning/simulation loop

getObservation

@AnyLogicInternalCodegenAPI
public abstract void getObservation(ROOT root,
                                    O observation)

This method must be defined in a subclass to get the data from root and write it to the fields of the given observation.
The method is called each step of reinforcement learning / AI test loop.

Specified by:: getObservation in interface ReinforcementLearningModel<ROOT extends Agent,O,A,C>
Parameters:: root - the root model agent; observation - data structure, to be filled from the simulation model and then sent to the Learning Agent (AI)

applyAction

@AnyLogicInternalCodegenAPI
public abstract void applyAction(ROOT root,
                                 A action)

This method must be defined in a subclass to get the data from the given action object and apply it to the model (root).
The method is called each step of reinforcement learning / AI test loop.

Specified by:: applyAction in interface ReinforcementLearningModel<ROOT extends Agent,O,A,C>
Parameters:: root - the root model agent; action - data structure, coming from the Learning Agent (AI) and then applied to the model

initDefaultRandomNumberGenerator

@AnyLogicInternalCodegenAPI
public void initDefaultRandomNumberGenerator(Engine engine)

This method should be overridden to initialize random number generator of the given engine
Default implementation set new random generation with random seed - for unique experiments

Parameters:: engine - the engine instance

createModel

public ROOT createModel()

Description copied from interface: ReinforcementLearningModel

Is called to obtain a new pre-initialized top-level agent. This method must be defined in a subclass. Note that the top-level agent should just be constructed in this method, as its create() and start() methods will be called later on.
This method should not be used to define parameters

Specified by:: createModel in interface ReinforcementLearningModel<ROOT extends Agent,O,A,C>

run

public void run()

Description copied from class: ExperimentCustom

Use ExperimentCustom.createEngine()

Specified by:: run in class ExperimentCustom

takeActionForModel

@AnyLogicInternalAPI
public void takeActionForModel(ROOT root)

Description copied from interface: LearningAgentModelInterface

This method is internal and shouldn't be called by user.
it may be removed/renamed in future.

Specified by:: takeActionForModel in interface LearningAgentModelInterface<ROOT extends Agent>

takeAction

public static void takeAction(Agent agent)

This method should be called from the user event associated with the experiment step to be performed. As an argument, pass any existing model agent. For example, when the event is located in the context of some agent, the following code should be used:
ExperimentReinforcementLearning.takeAction( this );.

This function will identify the passed 'agent' argument with the Main model agent ('root'), forcing all RL-related data processing (such as retrieving the observation data) to be done in the context of this agent.

Parameters:: agent - any existing agent

time

public double time()

Returns the current model (logical) time. Time stays constant during an event execution and is only advanced between events.

Returns:: current model time

time

public double time(TimeUnits units)

Returns the current model (logical) time. Time stays constant during an event execution and is only advanced between events.

Parameters:: units - the time units
Returns:: current model time

date

public java.util.Date date()

Returns the current model date with respect to the start time/date and the model time unit.

Returns:: the current model date

How can we improve this article?

Class ExperimentReinforcementLearning<ROOT extends Agent,​O,​A,​C>

Constructor Summary

Method Summary

Methods inherited from class com.anylogic.engine.ExperimentCustom

Methods inherited from class java.lang.Object

Constructor Detail

ExperimentReinforcementLearning

ExperimentReinforcementLearning

Method Detail

createRoot

createObservation

createAction

createConfiguration

applyConfiguration

checkEpisodeStopCondition

getObservation

applyAction

initDefaultRandomNumberGenerator

createModel

run

takeActionForModel

takeAction

time

time

date

Class ExperimentReinforcementLearning<ROOT extends Agent,O,A,C>