Faculty Profiles - HACHIYA Hirotaka

Published Papers - HACHIYA Hirotaka

Division display >> ／ All the affair displays1 - 63 of about 63

DCLMA: Deep correlation learning with multi-modal attention for visual-audio retrieval

Jiwei Zhang, Hirotaka Hachiya （Part： Last author )

Machine Learning with Applications ( Elsevier BV ) 100695 - 100695 2025.07 [Refereed]

DOI

FIRE-AD: frequency-dependent image reconstruction error for micro defect detection

Yuhei Nomura, Hirotaka Hachiya （Part： Last author )

Machine Vision and Applications 2025.07 [Refereed]

DOI

A Transformer-Based Fully Trainable Point Process

Hirotaka HACHIYA, Fumiya NISHIZAWA

IEICE Transactions on Information and Systems ( Institute of Electronics, Information and Communications Engineers (IEICE) ) E108.D ( 6 ) 583 - 592 2025.06

DOI

Enhancing semantic audio-visual representation learning with supervised multi-scale attention

Jiwei Zhang, Yi Yu, Suhua Tang, GuoJun Qi, Haiyuan Wu, Hirotaka Hachiya （Part： Last author )

Pattern Analysis and Applications ( Springer Science and Business Media LLC ) 28 ( 2 ) 2025.02 [Refereed]

DOI

Deep Supervised with Fine-grained Feature Fusion Network for Cross-modal Retrieval

Jiwei Zhang, Hirotaka Hachiya （Part： Last author )

International Conference on Artificial Intelligence in Information and Communication (ICAIIC) 2025.02 [Refereed]

Randomized Channel-Pass Mask for Channel-Wise Explanation of Black-Box Models

Hirotaka Hachiya, Daiki Nisawa （Part： Lead author )

Lecture Notes in Computer Science ( Springer Nature Singapore ) 454 - 468 2024.12

DOI

MLP-Mixer based surrogate model for seismic ground motion with spatial source and geometry parameters

Hirotaka Hachiya, Yuto Kuroki, Asako Iwaki, Takahiro Maeda, Naonori Ueda, Hiroyuki Fujiwara （Part： Lead author )

Proceedings of Asian Conference on Machine Learning (ACML) 2024.12 [Refereed]

Specular Surface Detection with Deep Static Specular Flow and Highlight

Hirotaka Hachiya, Yuto Yoshimura （Part： Lead author )

Machine Vision and Applications ( Springer Science and Business Media LLC ) 35 ( 6 ) 2024.09 [Refereed]

DOI

Permutation Dependent Feature Mixing in TSMixer for Multivariate Time Series Forecasting

Rikuto Yamazono, Hirotaka Hachiya （Part： Last author )

Proceedings of European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases (ECMLPKDD), 2024 2024.09 [Refereed]

Multi-class AUC maximization for imbalanced ordinal multi-stage tropical cyclone intensity change forecast

Hirotaka Hachiya, Hiroki Yoshida, Udai Shimada, Naonori Ueda （Part： Lead author )

Machine Learning with Applications ( Elsevier BV ) 17 100569 - 100569 2024.09 [Refereed]

DOI

A Multi‐Fusion Residual Attention U‐Net Using Temporal Information for Segmentation of Left Ventricular Structures in 2D Echocardiographic Videos

Kai Wang, Hirotaka Hachiya, Haiyuan Wu （Part： Last author )

International Journal of Imaging Systems and Technology ( Wiley ) 34 ( 4 ) 2024.07 [Refereed]

ABSTRACT

The interpretation of cardiac function using echocardiography requires a high level of diagnostic proficiency and years of experience. This study proposes a multi‐fusion residual attention U‐Net, MURAU‐Net, to construct automatic segmentation for evaluating cardiac function from echocardiographic video. MURAU‐Net has two benefits: (1) Multi‐fusion network to strengthen the links between spatial features. (2) Inter‐frame links can be established to augment the temporal coherence of sequential image data, thereby enhancing its continuity. To evaluate the effectiveness of the proposed method, we performed nine‐fold cross‐validation using CAMUS dataset. Among state‐of‐the‐art methods, MURAU‐Net achieves highly competitive score, for example, Dice similarity of 0.952 (ED phase) and 0.931 (ES phase) in , 0.966 (ED phase) and 0.957 (ES phase) in , and 0.901 (ED phase) and 0.917 (ES phase) in , respectively. It also achieved the Dice similarity of 0.9313 in the EchoNet‐Dynamic dataset for the overall left ventricle segmentation. In addition, we show MURAU‐Net can accurately segment multiclass cardiac ultrasound videos and output the animation of segmentation results using the original two‐chamber cardiac ultrasound dataset MUCO.

DOI

Set representative vector and its asymmetric attention-based transformation for heterogeneous set-to-set matching

Hirotaka Hachiya, Yuki Saito （Part： Lead author )

Neurocomputing 2024.04 [Refereed]

DOI

INTERPRETABLE DEEP INPAINTING BASED ON SUBSURFACE STRUCTURE DATA FOR SPATIAL INTERPOLATION OF SEISMIC MOTIONS

HACHIYA Hirotaka, TARASUKI Yuka, IWAKI Asako, MAEDA Takahiro, UEDA Naonori, FUJIWARA Hiroyuki （Part： Lead author )

Journal of Japan Association for Earthquake Engineering ( Japan Association for Earthquake Engineering ) 24 ( 5 ) 5_35 - 5_44 2024 [Refereed]

DOI

Frequency-Dependent Image Reconstruction Error for Micro Defect Detection

Yuhei Nomura, Hirotaka Hachiya （Part： Last author )

Proceedings of The 15th Asian Conference on Machine Learning (ACML2023) 2023.11 [Refereed]

Position-dependent partial convolutions for supervised spatial interpolation

Hirotaka Hachiya, Kotaro Nagayoshi, Asako Iwaki, Takahiro Maeda, Naonori Ueda, Hiroyuki Fujiwara （Part： Lead author )

Machine Learning with Applications ( Elsevier BV ) 100514 - 100514 2023.11 [Refereed]

DOI

Combining Static Specular Flow and Highlight with Deep Features for Specular Surface Detection

Hirotaka Hachiya, Yuta Yoshimura （Part： Lead author )

Proceedings of The 18th International Conference on Machine Vision Applications (MVA2023) 2023.07 [Refereed]

Encoder–decoder-based image transformation approach for integrating multiple spatial forecasts

Hirotaka Hachiya, Yusuke Masumoto, Atsushi Kudo, Naonori Ueda （Part： Lead author )

Machine Learning with Applications ( Elsevier BV ) 12 ( 100473 ) 1 - 11 2023.05 [Refereed]

DOI

Multistream-Based Marked Point Process With Decomposed Cumulative Hazard Functions

Hirotaka Hachiya, Sujun Hong （Part： Lead author )

Neural Computation ( MIT Press ) 35 ( 4 ) 699 - 726 2023.03 [Refereed]

Abstract

When applying a point process to a real-world problem, an appropriate intensity function model should be designed based on physical and mathematical prior knowledge. Recently, a fully trainable deep learning–based approach has been developed for temporal point processes. In this approach, a cumulative hazard function (CHF) capable of systematic computation of adaptive intensity function is modeled in a data-driven manner. However, in this approach, although many applications of point processes generate various kinds of information such as location, magnitude, and depth, the mark information of events is not considered. To overcome this limitation, we propose a fully trainable marked point process method for modeling decomposed CHFs for time and mark prediction using multistream deep neural networks. We demonstrate the effectiveness of the proposed method through experiments with synthetic and real-world event data.

DOI

Multi‐feature subspace representation network for person re‐identification via bird's‐eye view image

Jiwei Zhang, Haiyuan Wu, Qian Chen, Hirotaka Hachiya

Computer Animation and Virtual Worlds ( Wiley ) 2023.02 [Refereed]

DOI

Position-dependent partial convolutions for supervised spatial interpolation

Hirotaka Hachiya, Kotaro Nagayoshi, Asako Iwaki, Takahiro Maeda, Naonori Ueda, Hiroyuki Fujiwara （Part： Lead author )

Proceedings of The 14th Asian Conference on Machine Learning (ACML2022) 189 420 - 435 2022.12 [Refereed]

Study on Echocardiographic Image Segmentation Based on Attention U-Net

Kai Wang, Jiwei Zhang, Hirotaka Hachiya, Haiyuan Wu

2022 IEEE International Conference on Mechatronics and Automation (ICMA) ( IEEE ) 2022.08 [Refereed]

DOI

Encoder-decoder-based image transformation approach for integrating precipitation forecasts ACML

Hirotaka Hachiya, Yusuke Masumoto, Yuki Mori, Naonori Ueda （Part： Lead author )

Proceedings of the 13th Asian Conference on Machine Learning (ACML2021) 2021.11 [Refereed]

Multi-stream based Marked Point Process

Sujun Hong, Hirotaka Hachiya （Part： Corresponding author )

Proceedings of the 13th Asian Conference on Machine Learning (ACML2021) 2021.11 [Refereed]

Simulation of broad-band ground motions with consistent long-period and short-period components using the Wasserstein interpolation of acceleration envelopes

Tomohisa Okazaki, Hirotaka Hachiya, Asako Iwaki, Takahiro Maeda, Hiroyuki Fujiwara, Naonori Ueda

Geophysical Journal International ( OXFORD UNIV PRESS ) 227 ( 1 ) 333 - 349 2021.10 [Refereed]

Practical hybrid approaches for the simulation of broad-band ground motions often combine long-period and short-period waveforms synthesized by independent methods under different assumptions for different period ranges, which at times can lead to incompatible time histories and frequency properties. This study explores an approach that generates consistent broad-hand waveforms using past observation records, under the assumption that long-period waveforms can he obtained from physics-based simulations. Specifically, acceleration envelopes and Fourier amplitude spectra are transformed from long-period to short-period using machine learning methods, and they arc combined to produce a broad-band waveform. To effectively obtain the relationship of high-dimensional envelopes from limited amount of data, we (I) l'onnulate the problem as the conversion of probability distributions, which enables the introduction of a metric known as the Wasserstein distance, and (2) embed pairs of longperiod and short-period envelopes into a common latent space to improve the consistency of the entire waveform. An experimental application to a past earthquake demonstrates that the proposed method exhibits superior performance compared to existing methods as well as neural network approaches. In particular, the proposed method reproduces global properties in the time domain, which confirms the effectiveness of the embedding approach as well as the advantage of the Wasserstein distance as a measure of dissimilarity of the envelopes. This method serves as a novel machine learning approach that maintains consistency both in the time-domain and frequency-domain properties of waveforms.

DOI

Exchangeable Deep Neural Networks for Set-to-Set Matching and Learning

Yuki Saito, Takuma Nakamura, Hirotaka Hachiya, Kenji Fukumizu

Proceedings of the 16th European Conference on Computer Vision (ECCV2020) ( Springer International Publishing ) 626 - 646 2020.08 [Refereed]

DOI

Direct Multi-class AUC Maximization for Forecasting Rapidly Intensifying Tropical Cyclones

Hirotaka Hachiya, Shumpei Kurora, Udai Shimada, Naonori Ueda （Part： Lead author )

JpGU-AGU Joint Meeting 2020 2020.07 [Refereed]

Adaptive truncated residuals regression for fine-grained regression problems

Hirotaka Hachiya, Yu Yamamamoto, Kazuhiro Hirahara, Naonori Ueda （Part： Lead author )

Proceedings of the 11th Asian Conference on Machine Learning (ACML2019) ( ACML ) 2019.11 [Refereed]

Machine Learning Approach for Adaptive Integration of Multiple Relative Intensity Models toward Improved Earthquake Forecasts in Japan

Hirotaka Hachiya, Kazuro Hirahara, Naonori Ueda

International Union of Geodesy and Geophysics (IUGG2019) ( IUGG ) 2019.07 [Refereed]

Broadband ground-motion synthesis using embeddig machine learning

omohisa Okazaki, Hirotaka Hachiya, Naonori Ueda, Asako Iwaki, Takahiro Maeda, Hiroyuki Fujiwara

International Union of Geodesy and Geophysics (IUGG2019) 2019.05 [Refereed]

Machine learning approach for constraining the plausible ranges of frictional parameters on the Philippine Sea plate reproducing the historical sequences of the Nankai megaquakes

Hirotaka Hachiya, Yu Yamamamoto, Kazuhiro Hirahara, Atsushi Takahashi, Naonori Ueda

European Geosciences Union (EGU) ( EGU ) 2019.04 [Refereed]

Synthesis of Broadband Ground Motions Using Embedding and Neural Networks

Tomohisa Okazaki, Hirotaka Hachiya, Naonori Ueda, Asako Iwaki, Takahiro Maeda, Hiroyuki Fujiwara

European Geosciences Union (EGU) ( EGU ) 2019.04 [Refereed]

3D Faster R-CNNとレーザスキャンとの組み合わせによる特定物体の頑健な距離推定

八谷大岳、射手矢和真、中村恭之

計測自動制御学会論文集 55(1) 2019.01 [Refereed]

Distance estimation with 2.5D anchors and its application to robot navigation

Hirotaka Hachiya, Yuki Saito, Kazuma Iteya, Masaya Nomura, Takayuki Nakamura

ROBOMECH Journal ( Springer Nature ) 5 ( 1 ) 2018.12 [Refereed]

DOI

2.5D Faster R-CNN for Distance Estimation

Hirotaka Hachiya, Yuki Saito, Kazuma Iteya, Masaya Nomura, Takayuki Nakamura

IEEE International Conference on Systems, Man, and Cybernetics ( IEEE ) 2018.10 [Refereed]

Information-Maximization Clustering Based on Squared-Loss Mutual Information.

Masashi Sugiyama, Gang Niu, Makoto Yamada, Manabu Kimura, Hirotaka Hachiya

Neural Computation 26 ( 1 ) 84 - 131 2014 [Refereed]

DOI

Computationally Efficient Multi-Label Classification by Least-Squares Probabilistic Classifiers

Hyunha Nam, Hirotaka Hachiya, Masashi Sugiyama

IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS ( IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG ) E96D ( 8 ) 1871 - 1874 2013.08 [Refereed]

Multi-label classification allows a sample to belong to multiple classes simultaneously, which is often the case in real-world applications such as text categorization and image annotation. In multi-label scenarios, taking into account correlations among multiple labels can boost the classification accuracy. However, this makes classifier training more challenging because handling multiple labels induces a high-dimensional optimization problem. In this paper, we propose a scalable multi-label method based on the least-squares probabilistic classifier. Through experiments, we show the usefulness of our proposed method.

DOI

Feature Selection via l(1)-Penalized Squared-Loss Mutual Information

Wittawat Jitkrittum, Hirotaka Hachiya, Masashi Sugiyama

IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS ( IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG ) E96D ( 7 ) 1513 - 1524 2013.07 [Refereed]

Feature selection is a technique to screen out less important features. Many existing supervised feature selection algorithms use redundancy and relevancy as the main criteria to select features. However, feature interaction, potentially a key characteristic in real-world problems, has not received much attention. As an attempt to take feature interaction into account, we propose l(1)-LSMI, an l(1)-regularization based algorithm that maximizes a squared-loss variant of mutual information between selected features and outputs. Numerical results show that l(1)-LSMI performs well in handling redundancy, detecting non-linear dependency, and considering feature interaction.

DOI

Efficient Sample Reuse in Policy Gradients with Parameter-Based Exploration

Tingting Zhao, Hirotaka Hachiya, Voot Tangkaratt, Jun Morimoto, Masashi Sugiyama

NEURAL COMPUTATION ( MIT PRESS ) 25 ( 6 ) 1512 - 1547 2013.06 [Refereed]

The policy gradient approach is a flexible and powerful reinforcement learning method particularly for problems with continuous actions such as robot control. A common challenge is how to reduce the variance of policy gradient estimates for reliable policy updates. In this letter, we combine the following three ideas and give a highly effective policy gradient method: (1) policy gradients with parameter-based exploration, a recently proposed policy search method with low variance of gradient estimates; (2) an importance sampling technique, which allows us to reuse previously gathered data in a consistent way; and (3) an optimal baseline, which minimizes the variance of gradient estimates with their unbiasedness being maintained. For the proposed method, we give a theoretical analysis of the variance of gradient estimates and show its usefulness through extensive experiments.

Artist Agent: A Reinforcement Learning Approach to Automatic Stroke Generation in Oriental Ink Painting

Ning Xie, Hirotaka Hachiya, Masashi Sugiyama

IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS ( IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG ) E96D ( 5 ) 1134 - 1144 2013.05 [Refereed]

Oriental ink painting, called Sumi-e, is one of the most distinctive painting styles and has attracted artists around the world. Major challenges in Sumi-e simulation are to abstract complex scene information and reproduce smooth and natural brush strokes. To automatically generate such strokes, we propose to model the brush as a reinforcement learning agent, and let the agent learn the desired brush-trajectories by maximizing the sum of rewards in the policy search framework. To achieve better performance, we provide elaborate design of actions, states, and rewards specifically tailored for a Sumi-e agent. The effectiveness of our proposed approach is demonstrated through experiments on Sumi-e simulation.

DOI

Squared-loss Mutual Information Regularization

Gang Niu, Wittawat Jitkrittum, Bo Dai, Hirotaka Hachiya, Masashi Sugiyama

International Conference on Machine Learning (ICML2013) ( International Machine Learning Society (IMLS) ) 10--18 2013 [Refereed]

Relative Density-Ratio Estimation for Robust Distribution Comparison.

Makoto Yamada, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, Masashi Sugiyama

Neural Computation 25 ( 5 ) 1324 - 1370 2013 [Refereed]

DOI

Multi-Task Approach to Reinforcement Learning for Factored-State Markov Decision Problems

Jaak Simm, Masashi Sugiyama, Hirotaka Hachiya

IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS ( IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG ) E95D ( 10 ) 2426 - 2437 2012.10 [Refereed]

Reinforcement learning (RL) is a flexible framework for learning a decision rule in an unknown environment. However, a large number of samples are often required for finding a useful decision rule. To mitigate this problem, the concept of transfer learning has been employed to utilize knowledge obtained from similar RL tasks. However, most approaches developed so far are useful only in low-dimensional settings. In this paper, we propose a novel transfer learning idea that targets problems with high-dimensional states. Our idea is to transfer knowledge between state factors (e.g., interacting objects) within a single RL task. This allows the agent to learn the system dynamics of the target RL task with fewer data samples. The effectiveness of the proposed method is demonstrated through experiments.

DOI

Importance-weighted least-squares probabilistic classifier for covariate shift adaptation with application to human activity recognition

Hirotaka Hachiya, Masashi Sugiyama, Naonori Ueda

NEUROCOMPUTING ( ELSEVIER SCIENCE BV ) 80 93 - 101 2012.03 [Refereed]

Human activity recognition from accelerometer data (e.g., obtained by smart phones) is gathering a great deal of attention since it can be used for various purposes such as remote health-care. However, since collecting labeled data is bothersome for new users, it is desirable to utilize data obtained from existing users. In this paper, we formulate this adaptation problem as learning under covariate shift, and propose a cornputationally efficient probabilistic classification method based on adaptive importance sampling. The usefulness of the proposed method is demonstrated in real-world human activity recognition. (C) 2011 Elsevier B.V. All rights reserved.

DOI

Analysis and improvement of policy gradient estimation

Tingting Zhao, Hirotaka Hachiya, Gang Niu, Masashi Sugiyama

NEURAL NETWORKS ( PERGAMON-ELSEVIER SCIENCE LTD ) 26 118 - 129 2012.02 [Refereed]

Policy gradient is a useful model-free reinforcement learning approach, but it tends to suffer from instability of gradient estimates. In this paper, we analyze and improve the stability of policy gradient methods. We first prove that the variance of gradient estimates in the PGPE (policy gradients with parameter-based exploration) method is smaller than that of the classical REINFORCE method under a mild assumption. We then derive the optimal baseline for PGPE, which contributes to further reducing the variance. We also theoretically show that PGPE with the optimal baseline is more preferable than REINFORCE with the optimal baseline in terms of the variance of gradient estimates. Finally, we demonstrate the usefulness of the improved PGPE method through experiments. (c) 2011 Elsevier Ltd. All rights reserved.

DOI

COMPUTATIONALLY EFFICIENT MULTI-LABEL CLASSIFICATION BY LEAST-SQUARES PROBABILISTIC CLASSIFIER

Hyun Ha Nam, Hirotaka Hachiya, Masashi Sugiyama

2012 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP) ( IEEE ) 2077 - 2080 2012 [Refereed]

Multi-label classification allows a sample to belong to multiple classes simultaneously, which is often the case in real-world applications such as audio tagging, image annotation, video search, and text mining. In such a multi-label scenario, taking into account correlation between multiple labels can boost the classification accuracy. However, this in turn makes classifier training more challenging because handling multiple labels tends to induce a high-dimensional optimization problem. In this paper, we propose a highly scalable multilabel classifier based on a computationally efficient classification algorithm called the least-squares probabilistic classifier. Through experiments, we show the usefulness of our proposed method.

Reward-Weighted Regression with Sample Reuse for Direct Policy Search in Reinforcement Learning

Hirotaka Hachiya, Jan Peters, Masashi Sugiyama

NEURAL COMPUTATION ( MIT PRESS ) 23 ( 11 ) 2798 - 2832 2011.11 [Refereed]

Direct policy search is a promising reinforcement learning framework, in particular for controlling continuous, high-dimensional systems. Policy search often requires a large number of samples for obtaining a stable policy update estimator, and this is prohibitive when the sampling cost is expensive. In this letter, we extend an expectation-maximization-based policy search method so that previously collected samples can be efficiently reused. The usefulness of the proposed method, reward-weighted regression with sample reuse (R-3), is demonstrated through robot learning experiments. (This letter is an extended version of our earlier conference paper: Hachiya, Peters, & Sugiyama, 2009.)

DOI

On information-maximization clustering: Tuning parameter selection and analytic solution

Masashi Sugiyama, Makoto Yamada, Manabu Kimura, Hirotaka Hachiya

Proceedings of the 28th International Conference on Machine Learning, ICML 2011 65 - 72 2011 [Refereed]

Information-maximization clustering learns a probabilistic classifier in an unsupervised manner so that mutual information between feature vectors and cluster assignments is maximized. A notable advantage of this approach is that it only involves continuous optimization of model parameters, which is substantially easier to solve than discrete optimization of cluster assignments. However, existing methods still involve non-convex optimization problems, and therefore finding a good local optimal solution is not straightforward in practice. In this paper, we propose an alternative information-maximization clustering method based on a squared-loss variant of mutual information. This novel approach gives a clustering solution analytically in a computationally efficient way via kernel eigenvalue decomposition. Furthermore, we provide a practical model selection procedure that allows us to objectively optimize tuning parameters included in the kernel function. Through experiments, we demonstrate the usefulness of the proposed approach. Copyright 2011 by the author(s)/owner(s).

Least absolute policy iteration - A robust approach to value function approximation

Masashi Sugiyama, Hirotaka Hachiya, Hisashi Kashima, Tetsuro Mortmura

IEICE Transactions on Information and Systems E93-D ( 9 ) 2555 - 2565 2010.09 [Refereed]

Least-squares policy iteration is a useful reinforcement learning method in robotics due to its computational e?ciency. However, it tends to be sensitive to outliers in observed rewards. In this paper, we propose an alternative method that employs the absolute loss for enhancing robustness and reliability. The proposed method is formulated as a linear programming problem which can be solved eficiently by standard optimization software, so the computational advantage is not sacrificed for gaining robustness and reliability. We demonstrate the usefulness of the proposed approach through a simulated robot-control task. Copyright © 2010 The Institute of Electronics, Information and Communication Engineers.

DOI

Least Absolute Policy Iteration-A Robust Approach to Value Function Approximation

Masashi Sugiyama, Hirotaka Hachiya, Hisashi Kashima, Tetsuro Morimura

IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS ( IEICE-INST ELECTRONICS INFORMATION COMMUNICATIONS ENG ) E93D ( 9 ) 2555 - 2565 2010.09 [Refereed]

Least-squares policy iteration is a useful reinforcement learning method in robotics due to its computational efficiency. However, it tends to be sensitive to outliers in observed rewards. In this paper, we propose an alternative method that employs the absolute loss for enhancing robustness and reliability. The proposed method is formulated as a linear programming problem which can be solved efficiently by standard optimization software, so the computational advantage is not sacrificed for gaining robustness and reliability. We demonstrate the usefulness of the proposed approach through a simulated robot-control task.

DOI

Efficient exploration through active learning for value function approximation in reinforcement learning

Takayuki Akiyama, Hirotaka Hachiya, Masashi Sugiyama

NEURAL NETWORKS ( PERGAMON-ELSEVIER SCIENCE LTD ) 23 ( 5 ) 639 - 648 2010.06 [Refereed]

Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. The effectiveness of the proposed method, which we call active policy iteration (API), is demonstrated through simulations with a batting robot. (C) 2010 Elsevier Ltd. All rights reserved.

DOI

Parametric return density estimation for Reinforcement Learning

Tetsuro Morimura, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, Toshiyuki Tanaka

Proceedings of the 26th Conference on Uncertainty in Artificial Intelligence, UAI 2010 368 - 375 2010 [Refereed]

Most conventional Reinforcement Learning (RL) algorithms aim to optimize decision-making rules in terms of the expected re-turns. However, especially for risk management purposes, other risk-sensitive criteria such as the value-at-risk or the expected shortfall are sometimes preferred in real applications. Here, we describe a parametric method for estimating density of the returns, which allows us to handle various criteria in a unified manner. We first extend the Bellman equation for the conditional expected return to cover a conditional probability density of the returns. Then we derive an extension of the TD-learning algorithm for estimating the return densities in an unknown environment. As test instances, several parametric density estimation algorithms are presented for the Gaussian, Laplace, and skewed Laplace distributions. We show that these algorithms lead to risk-sensitive as well as robust RL paradigms through numerical experiments.

Least-squares conditional density estimation

Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, Daisuke Okanohara

IEICE Transactions on Information and Systems E93-D ( 3 ) 583 - 594 2010 [Refereed]

Estimating the conditional mean of an input-output relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation that is suitable for multi-dimensional continuous variables. The basic idea of the proposed method is to express the conditional density in terms of the density ratio and the ratio is directly estimated without going through density estimation. Experiments using benchmark and robot transition datasets illustrate the usefulness of the proposed approach. Copyright © 2010 The Institute of Electronics, Information and Communication Engineers.

DOI

Nonparametric return distribution approximation for reinforcement learning

Tetsuro Morimurat, Masashi Sugiyama, Hisashi Kashima, Hirotaka Hachiya, Toshiyuki Tanaka

ICML 2010 - Proceedings, 27th International Conference on Machine Learning 799 - 806 2010 [Refereed]

Standard Reinforcement Learning (RL) aims to optimize decision-making rules in terms of the expected return. However, especially for risk-management purposes, other criteria such as the expected shortfall are some-times preferred. Here, we describe a method of approximating the distribution of returns, which allows us to derive various kinds of information about the returns. We first show that the Bellman equation, which is a recursive formula for the expected return, can be extended to the cumulative return distribution. Then we derive a nonparametric return distribution estimator with particle smooth ing based on this extended Bellman equation. A key aspect of the proposed algorithm is to represent the recursion relation in the extended Bellman equation by a simple replacement procedure of particles associated with a state by using those of the successor state. We show that our algorithm leads to a risk-sensitive R.L paradigm. The usefulness of the proposed approach is demonstrated through numerical experiments. Copyright 2010 by the author(s)/owner(s).

Conditional density estimation via least-squares density ratio estimation

Masashi Sugiyama, Ichiro Takeuchi, Taiji Suzuki, Takafumi Kanamori, Hirotaka Hachiya, Daisuke Okanohara

Journal of Machine Learning Research 9 781 - 788 2010 [Refereed]

Estimating the conditional mean of an inputoutput relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation. Our basic idea is to express the conditional density in terms of the ratio of unconditional densities, and the ratio is directly estimated without going through density estimation. Experiments using benchmark and robot transition datasets illustrate the usefulness of the proposed approach. Copyright 2010 by the authors.

Feature Selection for Reinforcement Learning: Evaluating Implicit State-Reward Dependency via Conditional Mutual Information

Hirotaka Hachiya, Masashi Sugiyama

MACHINE LEARNING AND KNOWLEDGE DISCOVERY IN DATABASES, ECML PKDD 2010 ( SPRINGER-VERLAG BERLIN ) 6321 474 - 489 2010 [Refereed]

Model-free reinforcement learning (RL) is a machine learning approach to decision making in unknown environments. However, real-world RL tasks often involve high-dimensional state spaces, and then standard RL methods do not perform well. In this paper, we propose a new feature selection framework for coping with high dimensionality. Our proposed framework adopts conditional mutual information between return and state-feature sequences as a feature selection criterion, allowing the evaluation of implicit state-reward dependency. The conditional mutual information is approximated by a least-squares method, which results in a computationally efficient feature selection procedure. The usefulness of the proposed method is demonstrated on grid-world navigation problems.

Adaptive importance sampling for value function approximation in off-policy reinforcement learning

Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiayma, Jan Peters

Neural Networks 22 ( 10 ) 1399 - 1410 2009.12 [Refereed]

Off-policy reinforcement learning is aimed at efficiently using data samples gathered from a policy that is different from the currently optimized policy. A common approach is to use importance sampling techniques for compensating for the bias of value function estimators caused by the difference between the data-sampling policy and the target policy. However, existing off-policy methods often do not take the variance of the value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations. © 2009 Elsevier Ltd. All rights reserved.

DOI

Active Policy Iteration: Efficient Exploration through Active Learning for Value Function Approximation in Reinforcement Learning

Takayuki Akiyama, Hirotaka Hachiya, Masashi Sugiyama

21ST INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE (IJCAI-09), PROCEEDINGS ( IJCAI-INT JOINT CONF ARTIF INTELL ) 980 - 985 2009 [Refereed]

Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. We demonstrate the usefulness of the proposed method, named active policy iteration (API), through simulations with a batting robot.

Efficient Data Reuse in Value Function Approximation.

Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiyama, Jan Peters

ADPRL: 2009 IEEE SYMPOSIUM ON ADAPTIVE DYNAMIC PROGRAMMING AND REINFORCEMENT LEARNING ( IEEE ) 8 - + 2009 [Refereed]

Off-policy reinforcement learning is aimed at efficiently using data samples gathered from a policy that is different from the currently optimized policy. A common approach is to use importance sampling techniques for compensating for the bias of value function estimators caused by the difference between the data-sampling policy and the target policy. However, existing off-policy methods often do not take the variance of the value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. The usefulness of the proposed approach is demonstrated through simulated swing-up inverted-pendulum problem.

Efficient Sample Reuse in EM-Based Policy Search

Hirotaka Hachiya, Jan Peters, Masashi Sugiyama

MACHINE LEARNING AND KNOWLEDGE DISCOVERY IN DATABASES, PT I, ECML PKDD 2009 ( SPRINGER-VERLAG BERLIN ) 5781 469 - + 2009 [Refereed]

Direct policy search is a, promising reinforcement learning framework in particular for controlling in continuous, high-dimensional systems such as anthropomorphic robots. Policy search often requires a large number of samples for obtaining a stable policy update estimator due to its high flexibility. However, this is prohibitive when the sampling cost is expensive. Ill this paper, we extend all EM-based policy search method so that previously collected samples call be efficiently reused. The usefulness of the proposed method. called Reward-weighted Regression with. sample Reuse (R-3), is demonstrated through a robot learning experiment.

Least Absolute Policy Iteration for Robust Value Function Approximation

Masashi Sugiyama, Hirotaka Hachiya, Hisashi Kashima, Tetsuro Morimura

ICRA: 2009 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION, VOLS 1-7 ( IEEE ) 699 - + 2009 [Refereed]

Geodesic Gaussian kernels for value function approximation

Masashi Sugiyama, Hirotaka Hachiya, Christopher Towell, Sethu Vijayakumar

AUTONOMOUS ROBOTS ( SPRINGER ) 25 ( 3 ) 287 - 304 2008.10 [Refereed]

The least-squares policy iteration approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in simulated robot arm control and Khepera robot navigation.

DOI

Adaptive importance sampling with automatic model selection in value function approximation

Hirotaka Hachiya, Takayuki Akiyama, Masashi Sugiyama, Jan Peters

Proceedings of the National Conference on Artificial Intelligence 3 1351 - 1356 2008 [Refereed]

Off-policy reinforcement learning is aimed at efficiently reusing data samples gathered in the past, which is an essential problem for physically grounded AI as experiments are usually prohibitively expensive. A common approach is to use importance sampling techniques for compensating for the bias caused by the difference between data-sampling policies and the target policy. However, existing off-policy methods do not often take the variance of value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations. Copyright © 2008.

Value function approximation on non-linear manifolds for robot motor control

Masashi Sugiyama, Hirotaka Hachiya, Christopher Towell, Sethu Vijayakumar

PROCEEDINGS OF THE 2007 IEEE INTERNATIONAL CONFERENCE ON ROBOTICS AND AUTOMATION, VOLS 1-10 ( IEEE ) 1733 - + 2007 [Refereed]

The least squares approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in a simulated robot arm control and Khepera robot navigation.

　PREV-NEXT　

<HACHIYA Hirotaka>

Published Papers - HACHIYA Hirotaka