講演・口頭発表等 - 八谷　大岳

Improving Model-based Reinforcement Learning with Multitask Learning (バイオ情報学(BIO) Vol.2009-BIO-19)

Jaak Simm, Masashi Sugiyama, Hirotaka Hachiya

研究報告バイオ情報学（BIO）  2009年12月10日   情報処理学会

　概要を見る

We introduce an extension to standard reinforcement learning setting called observational RL (ORL) where additional observational information is available to the agent. This allows the agent to learn the system dynamics with fewer data samples, which is an essential feature for practical applications of RL methods. We show that ORL can be formulated as a multitask learning problem. A similarity-based and a component-based multitask learning methods are proposed for learning the transition probabilities of the ORL problem. The effectiveness of the proposed methods is evaluated in experiments of grid world and object lifting tasks.We introduce an extension to standard reinforcement learning setting called observational RL (ORL) where additional observational information is available to the agent. This allows the agent to learn the system dynamics with fewer data samples, which is an essential feature for practical applications of RL methods. We show that ORL can be formulated as a multitask learning problem. A similarity-based and a component-based multitask learning methods are proposed for learning the transition probabilities of the ORL problem. The effectiveness of the proposed methods is evaluated in experiments of grid world and object lifting tasks.

Conditional Density Estimation Based on Density Ratio Estimation

Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, Daisuke Okanohara

研究報告バイオ情報学（BIO）  2009年12月10日  

　概要を見る

Estimating the conditional mean of an input-output relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation that is suitable for multi-dimensional continuous variables. The basic idea of the proposed method is to express the conditional density in terms of the density ratio and the ratio is directly estimated without going through density estimation.Estimating the conditional mean of an input-output relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation that is suitable for multi-dimensional continuous variables. The basic idea of the proposed method is to express the conditional density in terms of the density ratio and the ratio is directly estimated without going through density estimation.

Statistical active learning for efficient value function approximation in reinforcement learning (ニューロコンピューティング)

Akiyama Takayuki, Hachiya Hirotaka, Sugiyama Masashi

電子情報通信学会技術研究報告. NC, ニューロコンピューティング  2009年03月04日   一般社団法人電子情報通信学会

　概要を見る

Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. The proposed method combined with LSPI is called active policy iteration (API). Through simulations we demonstrate the usefulness of API.

Adaptive importance sampling with automatic model selection in reward weighted regression (ニューロコンピューティング)

Hachiya Hirotaka, Peters Jan, Sugiyama Masashi

電子情報通信学会技術研究報告. NC, ニューロコンピューティング  2009年03月04日   一般社団法人電子情報通信学会

　概要を見る

Direct policy search is a promising reinforcement learning framework in particular for controlling in continuous, high-dimensional systems such as anthropomorphic robot. Policy search often requires a large number of samples for obtaining a stable policy update estimator due to its high flexibility. However, this is prohibitive when the sampling cost is expensive. In this paper, we extend an EM-based policy search method so that previously collected samples can be efficiently reused. The usefulness of the proposed method, called Reward-weighted Regression with sample Reuse (R^3), is demonstrated through a toy example.

Adaptive importance sampling with automatic model selection in value function approximation (ニューロコンピューティング)

Hachiya Hirotaka, Akiyama Takayuki, Sugiyama Masashi

電子情報通信学会技術研究報告. NC, ニューロコンピューティング  2007年12月15日   一般社団法人電子情報通信学会

　概要を見る

Off-policy reinforcement learning is aimed at efficiently reusing data samples gathered in the past. A common approach is to use importance sampling techniques for compensating for the bias caused by the difference between data-collecting policies and the target policy. However, existing off-policy methods do not often take the variance of value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a statistical machine learning theory.

Robot Control by Least-Squares Policy Iteration with Geodesic Gaussian Kernels

Hachiya Hirotaka, Sugiyama Masashi

人工知能学会全国大会論文集  2007年   人工知能学会