Reinforcement learning functions. More...

#include "xcs_rl.h"
#include "clset.h"
#include "ea.h"
#include "env.h"
#include "pa.h"
#include "param.h"
#include "perf.h"
#include "utils.h"

Include dependency graph for xcs_rl.c:

Functions
static double	xcs_rl_trial (struct XCSF xcsf, double error, const bool explore)
	Executes a reinforcement learning trial using a built-in environment.

double	xcs_rl_exp (struct XCSF *xcsf)
	Executes a reinforcement learning experiment.

double	xcs_rl_fit (struct XCSF xcsf, const double state, const int action, const double reward)
	Creates and updates an action set for a given (state, action, reward).

void	xcs_rl_init_trial (struct XCSF *xcsf)
	Initialises a reinforcement learning trial.

void	xcs_rl_end_trial (struct XCSF *xcsf)
	Frees memory used by a reinforcement learning trial.

void	xcs_rl_init_step (struct XCSF *xcsf)
	Initialises a step in a reinforcement learning trial.

void	xcs_rl_end_step (struct XCSF xcsf, const double state, const int action, const double reward)
	Ends a step in a reinforcement learning trial.

void	xcs_rl_update (struct XCSF xcsf, const double state, const int action, const double reward, const bool done)
	Provides reinforcement to the sets.

double	xcs_rl_error (struct XCSF *xcsf, const int action, const double reward, const bool done, const double max_p)
	Returns the reinforcement learning system prediction error.

int	xcs_rl_decision (struct XCSF xcsf, const double state)
	Selects an action to perform in a reinforcement learning problem.

Detailed Description

Reinforcement learning functions.

Author: Richard Preen rpree.nosp@m.n@gm.nosp@m.ail.c.nosp@m.om

Copyright: The Authors.

Date: 2015–2023.

A trial consists of one or more steps.

Definition in file xcs_rl.c.

Function Documentation

◆ xcs_rl_decision()

int xcs_rl_decision	(	struct XCSF *	xcsf,
		const double *	state
	)

Selects an action to perform in a reinforcement learning problem.

Constructs the match set and selects an action to perform.

Parameters

[in]	xcsf	The XCSF data structure.
[in]	state	The input state.

Returns: The selected action.

Definition at line 247 of file xcs_rl.c.

References clset_match(), pa_best_action(), pa_build(), pa_rand_action(), and rand_uniform().

Referenced by XCS::decision(), and xcs_rl_trial().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ xcs_rl_end_step()

void xcs_rl_end_step	(	struct XCSF *	xcsf,
		const double *	state,
		const int	action,
		const double	reward
	)

Ends a step in a reinforcement learning trial.

Parameters

[in]	xcsf	The XCSF data structure.
[in]	state	The current input state.
[in]	action	The current action.
[in]	reward	The current reward.

Definition at line 171 of file xcs_rl.c.

References clset_free(), and pa_val().

Referenced by XCS::end_step(), xcs_rl_fit(), and xcs_rl_trial().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ xcs_rl_end_trial()

void xcs_rl_end_trial ( struct XCSF * xcsf )

Frees memory used by a reinforcement learning trial.

Parameters

[in] xcsf The XCSF data structure.

Definition at line 145 of file xcs_rl.c.

References clset_free(), and clset_kill().

Referenced by XCS::end_trial(), xcs_rl_fit(), and xcs_rl_trial().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ xcs_rl_error()

double xcs_rl_error	(	struct XCSF *	xcsf,
		const int	action,
		const double	reward,
		const bool	done,
		const double	max_p
	)

Returns the reinforcement learning system prediction error.

Parameters

[in]	xcsf	The XCSF data structure.
[in]	action	The current action.
[in]	reward	The current reward.
[in]	done	Whether the environment is in a terminal state.
[in]	max_p	The maximum payoff in the environment.

Returns: The prediction error.

Definition at line 223 of file xcs_rl.c.

References pa_val().

Referenced by XCS::error(), and xcs_rl_trial().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ xcs_rl_exp()

double xcs_rl_exp ( struct XCSF * xcsf )

Executes a reinforcement learning experiment.

Parameters

[in] xcsf The XCSF data structure.

Returns: The mean number of steps to goal.

Definition at line 78 of file xcs_rl.c.

References perf_print(), and xcs_rl_trial().

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ xcs_rl_fit()

double xcs_rl_fit	(	struct XCSF *	xcsf,
		const double *	state,
		const int	action,
		const double	reward
	)

Creates and updates an action set for a given (state, action, reward).

Parameters

[in]	xcsf	The XCSF data structure.
[in]	state	The input state to match.
[in]	action	The selected action.
[in]	reward	The reward for having performed the action.

Returns: The prediction error.

Definition at line 104 of file xcs_rl.c.

References clset_match(), pa_build(), pa_val(), param_set_explore(), xcs_rl_end_step(), xcs_rl_end_trial(), xcs_rl_init_step(), xcs_rl_init_trial(), and xcs_rl_update().

Referenced by XCS::fit().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ xcs_rl_init_step()

void xcs_rl_init_step ( struct XCSF * xcsf )

Initialises a step in a reinforcement learning trial.

Parameters

[in] xcsf The XCSF data structure.

Definition at line 157 of file xcs_rl.c.

References clset_init().

Referenced by XCS::init_step(), xcs_rl_fit(), and xcs_rl_trial().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ xcs_rl_init_trial()

void xcs_rl_init_trial ( struct XCSF * xcsf )

Initialises a reinforcement learning trial.

Parameters

[in] xcsf The XCSF data structure.

Definition at line 126 of file xcs_rl.c.

References clset_init().

Referenced by XCS::init_trial(), xcs_rl_fit(), and xcs_rl_trial().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ xcs_rl_trial()

static double xcs_rl_trial	(	struct XCSF *	xcsf,
		double *	error,
		const bool	explore
	)

static

Executes a reinforcement learning trial using a built-in environment.

Parameters

[in]	xcsf	The XCSF data structure.
[out]	error	The mean system prediction error.
[in]	explore	Whether this is an exploration or exploitation trial.

Returns: Returns the accuracy for single-step problems and the number of steps taken to reach the goal for multi-step problems.

Definition at line 43 of file xcs_rl.c.

References env_execute(), env_get_state(), env_is_done(), env_max_payoff(), env_multistep(), env_reset(), param_set_explore(), xcs_rl_decision(), xcs_rl_end_step(), xcs_rl_end_trial(), xcs_rl_error(), xcs_rl_init_step(), xcs_rl_init_trial(), and xcs_rl_update().

Referenced by xcs_rl_exp().

Here is the call graph for this function:

Here is the caller graph for this function:

◆ xcs_rl_update()

void xcs_rl_update	(	struct XCSF *	xcsf,
		const double *	state,
		const int	action,
		const double	reward,
		const bool	done
	)

Provides reinforcement to the sets.

Creates the action set, updates the classifiers and runs the EA.

Parameters

[in]	xcsf	The XCSF data structure.
[in]	state	The input state.
[in]	action	The action selected.
[in]	reward	The reward from performing the action.
[in]	done	Whether the environment is in a terminal state.

Definition at line 192 of file xcs_rl.c.

References clset_action(), clset_update(), clset_validate(), ea(), and pa_best_val().

Referenced by XCS::update(), xcs_rl_fit(), and xcs_rl_trial().

Here is the call graph for this function:

Here is the caller graph for this function:

Functions

Detailed Description

Function Documentation

◆ xcs_rl_decision()

◆ xcs_rl_end_step()

◆ xcs_rl_end_trial()

◆ xcs_rl_error()

◆ xcs_rl_exp()

◆ xcs_rl_fit()

◆ xcs_rl_init_step()

◆ xcs_rl_init_trial()

◆ xcs_rl_trial()

◆ xcs_rl_update()