Package com.anylogic.engine
- java.lang.Object
-
- com.anylogic.engine.ExperimentCustom
-
- com.anylogic.engine.ExperimentReinforcementLearning<ROOT,O,A,C>
- Type Parameters:
ROOT
- root model agent typeO
- Observation data structure type, seegetObservation(Agent, Object)
A
- Action data structure type, seeapplyAction(Agent, Object)
C
- Configuration data structure type, seeapplyConfiguration(Agent, Object)
- All Implemented Interfaces:
LearningAgentModelInterface<ROOT>
,ReinforcementLearningModel<ROOT,O,A,C>
,java.io.Serializable
public abstract class ExperimentReinforcementLearning<ROOT extends Agent,O,A,C> extends ExperimentCustom implements ReinforcementLearningModel<ROOT,O,A,C>, LearningAgentModelInterface<ROOT>
Base class for Reinforcement Learning experiments.
This class serves as an interface between the model and the platform (Learning Platform) using Reinforcement Learning to train some Learning Agent which is based on that platform (e.g. artificial neural network). The experiment itself is platform-agnostic and speaks in general terms. Experiment may be used in two forms:
Below, we describe an example use: it may perform several simulation runs - the sequence of so called Episodes. The number of Episodes is determined by the Learning Platform. Each Episode contains the following logic:
1. Get the Episode Configuration (used for model initialization) from the Learning Agent (may be absent for models which don't require specific initialization).
2. Create root model agent and initialize it using the Configuration:
3. Start simulation:
3.1 Check the Stop condition (if
3.2 Get Observation from the model:
3.3 Pass the Observation to the Learning Agent and receive the Action (or alternative responses like Stop Episode or Stop learning)
3.4 Apply the Action to the model:
3.5 Run the model until some Stepping point, which should be defined by a user event or another trigger inside the model. To define this point, call
3.6 Repeat to 3.1
If your Learning Platform has finished learning, and you want to test your Learning Agent. To do that, add a block from platform-specific reinforcement learning library to your root (Main) model agent, fill in the properties of that block and ensure there is an event/code inside your model calling the code noted in the p.3.5 above. Then simply run any simulation experiment with your model.
Usage:
This class serves as an interface between the model and the platform (Learning Platform) using Reinforcement Learning to train some Learning Agent which is based on that platform (e.g. artificial neural network). The experiment itself is platform-agnostic and speaks in general terms. Experiment may be used in two forms:
- exported to the Learning Platform supported by AnyLogic,
- and used during simulation with an already trained Learning Agent (testing purpose) - together with the Learning Platform-specific connector block from the Reinforcement Learning Library.
Below, we describe an example use: it may perform several simulation runs - the sequence of so called Episodes. The number of Episodes is determined by the Learning Platform. Each Episode contains the following logic:
1. Get the Episode Configuration (used for model initialization) from the Learning Agent (may be absent for models which don't require specific initialization).
2. Create root model agent and initialize it using the Configuration:
applyConfiguration(Agent, Object)
3. Start simulation:
3.1 Check the Stop condition (if
true
, the Learning Platform will be notified about Episode termination condition)3.2 Get Observation from the model:
getObservation(Agent, Object)
3.3 Pass the Observation to the Learning Agent and receive the Action (or alternative responses like Stop Episode or Stop learning)
3.4 Apply the Action to the model:
applyAction(Agent, Object)
3.5 Run the model until some Stepping point, which should be defined by a user event or another trigger inside the model. To define this point, call
ExperimentReinforcementLearning.takeAction( this );
,
where this
is an existing agent containing the event.3.6 Repeat to 3.1
If your Learning Platform has finished learning, and you want to test your Learning Agent. To do that, add a block from platform-specific reinforcement learning library to your root (Main) model agent, fill in the properties of that block and ensure there is an event/code inside your model calling the code noted in the p.3.5 above. Then simply run any simulation experiment with your model.
Usage:
- Define observation, action and configuration data structure classes and corresponding methods (p.3).
- In order to define the moment when the reinforcement learning should take place, create an event or any other type action with the code described in p. 3.5 above.
- Author:
- AnyLogic North America, LLC https://anylogic.com
- See Also:
- Serialized Form
ExperimentReinforcementLearning() |
|
ExperimentReinforcementLearning(LearningAgentInterface<O,A,C> learningAgentInterface) |
|
Constructor | Description |
---|
abstract void |
applyAction(ROOT root,
A action) |
This method must be defined in a subclass to get the data from the given
action object and
apply it to the model (root ).The method is called each step of reinforcement learning / AI test loop. |
abstract void |
applyConfiguration(ROOT root,
C configuration) |
This method must be defined in a subclass to get the data from the given
configuration object and
apply it as the initial setup to the model (root ).The method is called only once per the whole model run - at the beginning. |
boolean |
checkEpisodeStopCondition(ROOT root) |
This method may be defined in a subclass to check the additional stop condition of the Episode.
|
abstract A |
createAction() |
This method must be defined in a subclass - just to create new empty Action object
|
abstract C |
createConfiguration() |
This method must be defined in a subclass - just to create new empty Configuration object
|
ROOT |
createModel() |
Is called to obtain a new pre-initialized top-level agent.
|
abstract O |
createObservation() |
This method must be defined in a subclass - just to create new empty Observation object
|
abstract ROOT |
createRoot(Engine engine) |
Is called to obtain a new top-level agent.
|
java.util.Date |
date() |
Returns the current model date with respect to the start time/date and the
model time unit.
|
abstract void |
getObservation(ROOT root,
O observation) |
This method must be defined in a subclass to get the data from
root and write it to the fields of the given observation .The method is called each step of reinforcement learning / AI test loop. |
void |
initDefaultRandomNumberGenerator(Engine engine) |
This method should be overridden to initialize random number generator of the given engine
Default implementation set new random generation with random seed - for unique experiments |
void |
run() |
|
static void |
takeAction(Agent agent) |
This method should be called from the user event associated with the experiment step to be performed.
|
void |
takeActionForModel(ROOT root) |
This method is internal and shouldn't be called by user.
it may be removed/renamed in future. |
double |
time() |
Returns the current model (logical) time.
|
double |
time(TimeUnits units) |
Returns the current model (logical) time.
|
Modifier and Type | Method | Description |
---|
createEngine, getCommandLineArguments, onError, onError, setCommandLineArguments_xjal, setupEngine_xjal
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
public ExperimentReinforcementLearning()
public ExperimentReinforcementLearning(LearningAgentInterface<O,A,C> learningAgentInterface)
@AnyLogicInternalCodegenAPI public abstract ROOT createRoot(Engine engine)
Is called to obtain a new top-level agent.
This method must be defined in a subclass. Note that
the top-level agent should just be constructed in this method, as
its create() and start() methods will be called later on.
This method should not be used to define parameters
This method should not be used to define parameters
- Parameters:
engine
- the simulation engine that will simulate the model
@AnyLogicInternalCodegenAPI public abstract O createObservation()
This method must be defined in a subclass - just to create new empty Observation object
- Specified by:
createObservation
in interfaceReinforcementLearningModel<ROOT extends Agent,O,A,C>
- Returns:
- new empty Observation object
@AnyLogicInternalCodegenAPI public abstract A createAction()
This method must be defined in a subclass - just to create new empty Action object
- Specified by:
createAction
in interfaceReinforcementLearningModel<ROOT extends Agent,O,A,C>
- Returns:
- new empty Action object
@AnyLogicInternalCodegenAPI public abstract C createConfiguration()
This method must be defined in a subclass - just to create new empty Configuration object
- Specified by:
createConfiguration
in interfaceReinforcementLearningModel<ROOT extends Agent,O,A,C>
- Returns:
- new empty Configuration object
@AnyLogicInternalCodegenAPI public abstract void applyConfiguration(ROOT root, C configuration)
This method must be defined in a subclass to get the data from the given
The method is called only once per the whole model run - at the beginning.
configuration
object and
apply it as the initial setup to the model (root
).The method is called only once per the whole model run - at the beginning.
- Specified by:
applyConfiguration
in interfaceReinforcementLearningModel<ROOT extends Agent,O,A,C>
- Parameters:
root
- the root model agentaction
- data structure, coming from the Reinforcement Learning platform and then applied to the model
@AnyLogicInternalCodegenAPI public boolean checkEpisodeStopCondition(ROOT root)
This method may be defined in a subclass to check the additional stop condition of the Episode.
Should return
true
to request stopping the Episode (e.g. when the model falls into some
undesired terminal state which doesn't allow further training or testing of Learning Agent).- Specified by:
checkEpisodeStopCondition
in interfaceReinforcementLearningModel<ROOT extends Agent,O,A,C>
- Returns:
true
to stop the model,false
to continue learning/simulation loop
@AnyLogicInternalCodegenAPI public abstract void getObservation(ROOT root, O observation)
This method must be defined in a subclass to get the data from
The method is called each step of reinforcement learning / AI test loop.
root
and write it to the fields of the given observation
.The method is called each step of reinforcement learning / AI test loop.
- Specified by:
getObservation
in interfaceReinforcementLearningModel<ROOT extends Agent,O,A,C>
- Parameters:
root
- the root model agentobservation
- data structure, to be filled from the simulation model and then sent to the Learning Agent (AI)
@AnyLogicInternalCodegenAPI public abstract void applyAction(ROOT root, A action)
This method must be defined in a subclass to get the data from the given
The method is called each step of reinforcement learning / AI test loop.
action
object and
apply it to the model (root
).The method is called each step of reinforcement learning / AI test loop.
- Specified by:
applyAction
in interfaceReinforcementLearningModel<ROOT extends Agent,O,A,C>
- Parameters:
root
- the root model agentaction
- data structure, coming from the Learning Agent (AI) and then applied to the model
@AnyLogicInternalCodegenAPI public void initDefaultRandomNumberGenerator(Engine engine)
This method should be overridden to initialize random number generator of the given engine
Default implementation set new random generation with random seed - for unique experiments
Default implementation set new random generation with random seed - for unique experiments
- Parameters:
engine
- the engine instance
public ROOT createModel()
Description copied from interface:
ReinforcementLearningModel
Is called to obtain a new pre-initialized top-level agent.
This method must be defined in a subclass. Note that
the top-level agent should just be constructed in this method, as
its create() and start() methods will be called later on.
This method should not be used to define parameters
This method should not be used to define parameters
- Specified by:
createModel
in interfaceReinforcementLearningModel<ROOT extends Agent,O,A,C>
public void run()
Description copied from class:
ExperimentCustom
- Specified by:
run
in classExperimentCustom
@AnyLogicInternalAPI public void takeActionForModel(ROOT root)
Description copied from interface:
LearningAgentModelInterface
This method is internal and shouldn't be called by user.
it may be removed/renamed in future.
it may be removed/renamed in future.
- Specified by:
takeActionForModel
in interfaceLearningAgentModelInterface<ROOT extends Agent>
public static void takeAction(Agent agent)
This method should be called from the user event associated with the experiment step to be performed.
As an argument, pass any existing model agent. For example, when the event is located in the context of some agent,
the following code should be used:
This function will identify the passed '
ExperimentReinforcementLearning.takeAction( this );
.This function will identify the passed '
agent
' argument with the Main model agent ('root'),
forcing all RL-related data processing (such as retrieving the observation data) to be done in the context of this agent.- Parameters:
agent
- any existing agent
public double time()
Returns the current model (logical) time. Time stays constant during
an event execution and is only advanced between events.
- Returns:
- current model time
public double time(TimeUnits units)
Returns the current model (logical) time. Time stays constant during
an event execution and is only advanced between events.
- Parameters:
units
- the time units- Returns:
- current model time
public java.util.Date date()
Returns the current model date with respect to the start time/date and the
model time unit.
- Returns:
- the current model date
-
How can we improve this article?
-