Faculty Profiles - HACHIYA Hirotaka

Published Papers - HACHIYA Hirotaka

Division display41 - 60 of about 62／ All the affair displays >>

Multi-Task Approach to Reinforcement Learning for Factored-State Markov Decision Problems

Jaak Simm, Masashi Sugiyama, Hirotaka Hachiya

IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS ( IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG ) E95D ( 10 ) 2426 - 2437 2012.10 [Refereed]

Reinforcement learning (RL) is a flexible framework for learning a decision rule in an unknown environment. However, a large number of samples are often required for finding a useful decision rule. To mitigate this problem, the concept of transfer learning has been employed to utilize knowledge obtained from similar RL tasks. However, most approaches developed so far are useful only in low-dimensional settings. In this paper, we propose a novel transfer learning idea that targets problems with high-dimensional states. Our idea is to transfer knowledge between state factors (e.g., interacting objects) within a single RL task. This allows the agent to learn the system dynamics of the target RL task with fewer data samples. The effectiveness of the proposed method is demonstrated through experiments.

DOI

Importance-weighted least-squares probabilistic classifier for covariate shift adaptation with application to human activity recognition

Hirotaka Hachiya, Masashi Sugiyama, Naonori Ueda

NEUROCOMPUTING ( ELSEVIER SCIENCE BV ) 80 93 - 101 2012.03 [Refereed]

Human activity recognition from accelerometer data (e.g., obtained by smart phones) is gathering a great deal of attention since it can be used for various purposes such as remote health-care. However, since collecting labeled data is bothersome for new users, it is desirable to utilize data obtained from existing users. In this paper, we formulate this adaptation problem as learning under covariate shift, and propose a cornputationally efficient probabilistic classification method based on adaptive importance sampling. The usefulness of the proposed method is demonstrated in real-world human activity recognition. (C) 2011 Elsevier B.V. All rights reserved.

DOI

Analysis and improvement of policy gradient estimation

Tingting Zhao, Hirotaka Hachiya, Gang Niu, Masashi Sugiyama

NEURAL NETWORKS ( PERGAMON-ELSEVIER SCIENCE LTD ) 26 118 - 129 2012.02 [Refereed]

Policy gradient is a useful model-free reinforcement learning approach, but it tends to suffer from instability of gradient estimates. In this paper, we analyze and improve the stability of policy gradient methods. We first prove that the variance of gradient estimates in the PGPE (policy gradients with parameter-based exploration) method is smaller than that of the classical REINFORCE method under a mild assumption. We then derive the optimal baseline for PGPE, which contributes to further reducing the variance. We also theoretically show that PGPE with the optimal baseline is more preferable than REINFORCE with the optimal baseline in terms of the variance of gradient estimates. Finally, we demonstrate the usefulness of the improved PGPE method through experiments. (c) 2011 Elsevier Ltd. All rights reserved.

DOI

COMPUTATIONALLY EFFICIENT MULTI-LABEL CLASSIFICATION BY LEAST-SQUARES PROBABILISTIC CLASSIFIER

Hyun Ha Nam, Hirotaka Hachiya, Masashi Sugiyama

2012 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP) ( IEEE ) 2077 - 2080 2012 [Refereed]

Multi-label classification allows a sample to belong to multiple classes simultaneously, which is often the case in real-world applications such as audio tagging, image annotation, video search, and text mining. In such a multi-label scenario, taking into account correlation between multiple labels can boost the classification accuracy. However, this in turn makes classifier training more challenging because handling multiple labels tends to induce a high-dimensional optimization problem. In this paper, we propose a highly scalable multilabel classifier based on a computationally efficient classification algorithm called the least-squares probabilistic classifier. Through experiments, we show the usefulness of our proposed method.

Reward-Weighted Regression with Sample Reuse for Direct Policy Search in Reinforcement Learning

Hirotaka Hachiya, Jan Peters, Masashi Sugiyama

NEURAL COMPUTATION ( MIT PRESS ) 23 ( 11 ) 2798 - 2832 2011.11 [Refereed]

Direct policy search is a promising reinforcement learning framework, in particular for controlling continuous, high-dimensional systems. Policy search often requires a large number of samples for obtaining a stable policy update estimator, and this is prohibitive when the sampling cost is expensive. In this letter, we extend an expectation-maximization-based policy search method so that previously collected samples can be efficiently reused. The usefulness of the proposed method, reward-weighted regression with sample reuse (R-3), is demonstrated through robot learning experiments. (This letter is an extended version of our earlier conference paper: Hachiya, Peters, & Sugiyama, 2009.)

DOI

On information-maximization clustering: Tuning parameter selection and analytic solution

Masashi Sugiyama, Makoto Yamada, Manabu Kimura, Hirotaka Hachiya

Proceedings of the 28th International Conference on Machine Learning, ICML 2011 65 - 72 2011 [Refereed]

Information-maximization clustering learns a probabilistic classifier in an unsupervised manner so that mutual information between feature vectors and cluster assignments is maximized. A notable advantage of this approach is that it only involves continuous optimization of model parameters, which is substantially easier to solve than discrete optimization of cluster assignments. However, existing methods still involve non-convex optimization problems, and therefore finding a good local optimal solution is not straightforward in practice. In this paper, we propose an alternative information-maximization clustering method based on a squared-loss variant of mutual information. This novel approach gives a clustering solution analytically in a computationally efficient way via kernel eigenvalue decomposition. Furthermore, we provide a practical model selection procedure that allows us to objectively optimize tuning parameters included in the kernel function. Through experiments, we demonstrate the usefulness of the proposed approach. Copyright 2011 by the author(s)/owner(s).

Least absolute policy iteration - A robust approach to value function approximation

Masashi Sugiyama, Hirotaka Hachiya, Hisashi Kashima, Tetsuro Mortmura

IEICE Transactions on Information and Systems E93-D ( 9 ) 2555 - 2565 2010.09 [Refereed]

Least-squares policy iteration is a useful reinforcement learning method in robotics due to its computational e?ciency. However, it tends to be sensitive to outliers in observed rewards. In this paper, we propose an alternative method that employs the absolute loss for enhancing robustness and reliability. The proposed method is formulated as a linear programming problem which can be solved eficiently by standard optimization software, so the computational advantage is not sacrificed for gaining robustness and reliability. We demonstrate the usefulness of the proposed approach through a simulated robot-control task. Copyright © 2010 The Institute of Electronics, Information and Communication Engineers.

DOI

Least Absolute Policy Iteration-A Robust Approach to Value Function Approximation

Masashi Sugiyama, Hirotaka Hachiya, Hisashi Kashima, Tetsuro Morimura

IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS ( IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG ) E93D ( 9 ) 2555 - 2565 2010.09 [Refereed]

Least-squares policy iteration is a useful reinforcement learning method in robotics due to its computational efficiency. However, it tends to be sensitive to outliers in observed rewards. In this paper, we propose an alternative method that employs the absolute loss for enhancing robustness and reliability. The proposed method is formulated as a linear programming problem which can be solved efficiently by standard optimization software, so the computational advantage is not sacrificed for gaining robustness and reliability. We demonstrate the usefulness of the proposed approach through a simulated robot-control task.

DOI

Efficient exploration through active learning for value function approximation in reinforcement learning

Takayuki Akiyama, Hirotaka Hachiya, Masashi Sugiyama

NEURAL NETWORKS ( PERGAMON-ELSEVIER SCIENCE LTD ) 23 ( 5 ) 639 - 648 2010.06 [Refereed]

Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. The effectiveness of the proposed method, which we call active policy iteration (API), is demonstrated through simulations with a batting robot. (C) 2010 Elsevier Ltd. All rights reserved.

DOI

Parametric return density estimation for Reinforcement Learning

Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, Toshiyuki Tanaka

Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence, UAI 2010 368 - 375 2010 [Refereed]

Most conventional Reinforcement Learning (RL) algorithms aim to optimize decision-making rules in terms of the expected re-turns. However, especially for risk management purposes, other risk-sensitive criteria such as the value-at-risk or the expected shortfall are sometimes preferred in real applications. Here, we describe a parametric method for estimating density of the returns, which allows us to handle various criteria in a unified manner. We first extend the Bellman equation for the conditional expected return to cover a conditional probability density of the returns. Then we derive an extension of the TD-learning algorithm for estimating the return densities in an unknown environment. As test instances, several parametric density estimation algorithms are presented for the Gaussian, Laplace, and skewed Laplace distributions. We show that these algorithms lead to risk-sensitive as well as robust RL paradigms through numerical experiments.

Least-squares conditional density estimation

Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, Daisuke Okanohara

IEICE Transactions on Information and Systems E93-D ( 3 ) 583 - 594 2010 [Refereed]

Estimating the conditional mean of an input-output relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation that is suitable for multi-dimensional continuous variables. The basic idea of the proposed method is to express the conditional density in terms of the density ratio and the ratio is directly estimated without going through density estimation. Experiments using benchmark and robot transition datasets illustrate the usefulness of the proposed approach. Copyright © 2010 The Institute of Electronics, Information and Communication Engineers.

DOI

Nonparametric return distribution approximation for reinforcement learning

Tetsuro Morimurat, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, Toshiyuki Tanaka

ICML 2010 - Proceedings, 27th International Conference on Machine Learning 799 - 806 2010 [Refereed]

Standard Reinforcement Learning (RL) aims to optimize decision-making rules in terms of the expected return. However, especially for risk-management purposes, other criteria such as the expected shortfall are some-times preferred. Here, we describe a method of approximating the distribution of returns, which allows us to derive various kinds of information about the returns. We first show that the Bellman equation, which is a recursive formula for the expected return, can be extended to the cumulative return distribution. Then we derive a nonparametric return distribution estimator with particle smooth ing based on this extended Bellman equation. A key aspect of the proposed algorithm is to represent the recursion relation in the extended Bellman equation by a simple replacement procedure of particles associated with a state by using those of the successor state. We show that our algorithm leads to a risk-sensitive R.L paradigm. The usefulness of the proposed approach is demonstrated through numerical experiments. Copyright 2010 by the author(s)/owner(s).

Conditional density estimation via least-squares density ratio estimation

Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, Daisuke Okanohara

Journal of Machine Learning Research 9 781 - 788 2010 [Refereed]

Estimating the conditional mean of an inputoutput relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation. Our basic idea is to express the conditional density in terms of the ratio of unconditional densities, and the ratio is directly estimated without going through density estimation. Experiments using benchmark and robot transition datasets illustrate the usefulness of the proposed approach. Copyright 2010 by the authors.

Feature Selection for Reinforcement Learning: Evaluating Implicit State-Reward Dependency via Conditional Mutual Information

Hirotaka Hachiya, Masashi Sugiyama

MACHINE LEARNING AND KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2010 ( SPRINGER-VERLAG BERLIN ) 6321 474 - 489 2010 [Refereed]

Model-free reinforcement learning (RL) is a machine learning approach to decision making in unknown environments. However, real-world RL tasks often involve high-dimensional state spaces, and then standard RL methods do not perform well. In this paper, we propose a new feature selection framework for coping with high dimensionality. Our proposed framework adopts conditional mutual information between return and state-feature sequences as a feature selection criterion, allowing the evaluation of implicit state-reward dependency. The conditional mutual information is approximated by a least-squares method, which results in a computationally efficient feature selection procedure. The usefulness of the proposed method is demonstrated on grid-world navigation problems.

Adaptive importance sampling for value function approximation in off-policy reinforcement learning

Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiayma, Jan Peters

Neural Networks 22 ( 10 ) 1399 - 1410 2009.12 [Refereed]

Off-policy reinforcement learning is aimed at efficiently using data samples gathered from a policy that is different from the currently optimized policy. A common approach is to use importance sampling techniques for compensating for the bias of value function estimators caused by the difference between the data-sampling policy and the target policy. However, existing off-policy methods often do not take the variance of the value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations. © 2009 Elsevier Ltd. All rights reserved.

DOI

Active Policy Iteration: Efficient Exploration through Active Learning for Value Function Approximation in Reinforcement Learning

Takayuki Akiyama, Hirotaka Hachiya, Masashi Sugiyama

21ST INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE (IJCAI-09), PROCEEDINGS ( IJCAI-INT JOINT CONF ARTIF INTELL ) 980 - 985 2009 [Refereed]

Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. We demonstrate the usefulness of the proposed method, named active policy iteration (API), through simulations with a batting robot.

Efficient Data Reuse in Value Function Approximation.

Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiyama, Jan Peters

ADPRL: 2009 IEEE SYMPOSIUM ON ADAPTIVE DYNAMIC PROGRAMMING AND REINFORCEMENT LEARNING ( IEEE ) 8 - + 2009 [Refereed]

Off-policy reinforcement learning is aimed at efficiently using data samples gathered from a policy that is different from the currently optimized policy. A common approach is to use importance sampling techniques for compensating for the bias of value function estimators caused by the difference between the data-sampling policy and the target policy. However, existing off-policy methods often do not take the variance of the value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. The usefulness of the proposed approach is demonstrated through simulated swing-up inverted-pendulum problem.

Efficient Sample Reuse in EM-Based Policy Search

Hirotaka Hachiya, Jan Peters, Masashi Sugiyama

MACHINE LEARNING AND KNOWLEDGE DISCOVERY IN DATABASES, PT I, ECML PKDD 2009 ( SPRINGER-VERLAG BERLIN ) 5781 469 - + 2009 [Refereed]

Direct policy search is a, promising reinforcement learning framework in particular for controlling in continuous, high-dimensional systems such as anthropomorphic robots. Policy search often requires a large number of samples for obtaining a stable policy update estimator due to its high flexibility. However, this is prohibitive when the sampling cost is expensive. Ill this paper, we extend all EM-based policy search method so that previously collected samples call be efficiently reused. The usefulness of the proposed method. called Reward-weighted Regression with. sample Reuse (R-3), is demonstrated through a robot learning experiment.

Least Absolute Policy Iteration for Robust Value Function Approximation

Masashi Sugiyama, Hirotaka Hachiya, Hisashi Kashima, Tetsuro Morimura

ICRA: 2009 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION, VOLS 1-7 ( IEEE ) 699 - + 2009 [Refereed]

Geodesic Gaussian kernels for value function approximation

Masashi Sugiyama, Hirotaka Hachiya, Christopher Towell, Sethu Vijayakumar

AUTONOMOUS ROBOTS ( SPRINGER ) 25 ( 3 ) 287 - 304 2008.10 [Refereed]

The least-squares policy iteration approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in simulated robot arm control and Khepera robot navigation.

DOI

　 PREV - NEXT 　

<HACHIYA Hirotaka>

Published Papers - HACHIYA Hirotaka