Published Papers - HACHIYA Hirotaka

Adaptive importance sampling with automatic model selection in value function approximation

Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiyama, Jan Peters

Proceedings of the National Conference on Artificial Intelligence   3   1351 - 1356   2008  [Refereed]

　View Summary

Off-policy reinforcement learning is aimed at efficiently reusing data samples gathered in the past, which is an essential problem for physically grounded AI as experiments are usually prohibitively expensive. A common approach is to use importance sampling techniques for compensating for the bias caused by the difference between data-sampling policies and the target policy. However, existing off-policy methods do not often take the variance of value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations. Copyright © 2008.

Value function approximation on non-linear manifolds for robot motor control

Masashi Sugiyama, Hirotaka Hachiya, Christopher Towell, Sethu Vijayakumar

PROCEEDINGS OF THE 2007 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION, VOLS 1-10 ( IEEE )    1733 - +   2007  [Refereed]

　View Summary

The least squares approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in a simulated robot arm control and Khepera robot navigation.