Abstract

When applying a point process to a real-world problem, an appropriate intensity function model should be designed based on physical and mathematical prior knowledge. Recently, a fully trainable deep learning–based approach has been developed for temporal point processes. In this approach, a cumulative hazard function (CHF) capable of systematic computation of adaptive intensity function is modeled in a data-driven manner. However, in this approach, although many applications of point processes generate various kinds of information such as location, magnitude, and depth, the mark information of events is not considered. To overcome this limitation, we propose a fully trainable marked point process method for modeling decomposed CHFs for time and mark prediction using multistream deep neural networks. We demonstrate the effectiveness of the proposed method through experiments with synthetic and real-world event data.

Practical hybrid approaches for the simulation of broad-band ground motions often combine long-period and short-period waveforms synthesized by independent methods under different assumptions for different period ranges, which at times can lead to incompatible time histories and frequency properties. This study explores an approach that generates consistent broad-hand waveforms using past observation records, under the assumption that long-period waveforms can he obtained from physics-based simulations. Specifically, acceleration envelopes and Fourier amplitude spectra are transformed from long-period to short-period using machine learning methods, and they arc combined to produce a broad-band waveform. To effectively obtain the relationship of high-dimensional envelopes from limited amount of data, we (I) l'onnulate the problem as the conversion of probability distributions, which enables the introduction of a metric known as the Wasserstein distance, and (2) embed pairs of longperiod and short-period envelopes into a common latent space to improve the consistency of the entire waveform. An experimental application to a past earthquake demonstrates that the proposed method exhibits superior performance compared to existing methods as well as neural network approaches. In particular, the proposed method reproduces global properties in the time domain, which confirms the effectiveness of the embedding approach as well as the advantage of the Wasserstein distance as a measure of dissimilarity of the envelopes. This method serves as a novel machine learning approach that maintains consistency both in the time-domain and frequency-domain properties of waveforms.

Multi-label classification allows a sample to belong to multiple classes simultaneously, which is often the case in real-world applications such as text categorization and image annotation. In multi-label scenarios, taking into account correlations among multiple labels can boost the classification accuracy. However, this makes classifier training more challenging because handling multiple labels induces a high-dimensional optimization problem. In this paper, we propose a scalable multi-label method based on the least-squares probabilistic classifier. Through experiments, we show the usefulness of our proposed method.

Feature selection is a technique to screen out less important features. Many existing supervised feature selection algorithms use redundancy and relevancy as the main criteria to select features. However, feature interaction, potentially a key characteristic in real-world problems, has not received much attention. As an attempt to take feature interaction into account, we propose l(1)-LSMI, an l(1)-regularization based algorithm that maximizes a squared-loss variant of mutual information between selected features and outputs. Numerical results show that l(1)-LSMI performs well in handling redundancy, detecting non-linear dependency, and considering feature interaction.

The policy gradient approach is a flexible and powerful reinforcement learning method particularly for problems with continuous actions such as robot control. A common challenge is how to reduce the variance of policy gradient estimates for reliable policy updates. In this letter, we combine the following three ideas and give a highly effective policy gradient method: (1) policy gradients with parameter-based exploration, a recently proposed policy search method with low variance of gradient estimates; (2) an importance sampling technique, which allows us to reuse previously gathered data in a consistent way; and (3) an optimal baseline, which minimizes the variance of gradient estimates with their unbiasedness being maintained. For the proposed method, we give a theoretical analysis of the variance of gradient estimates and show its usefulness through extensive experiments.

Oriental ink painting, called Sumi-e, is one of the most distinctive painting styles and has attracted artists around the world. Major challenges in Sumi-e simulation are to abstract complex scene information and reproduce smooth and natural brush strokes. To automatically generate such strokes, we propose to model the brush as a reinforcement learning agent, and let the agent learn the desired brush-trajectories by maximizing the sum of rewards in the policy search framework. To achieve better performance, we provide elaborate design of actions, states, and rewards specifically tailored for a Sumi-e agent. The effectiveness of our proposed approach is demonstrated through experiments on Sumi-e simulation.

Reinforcement learning (RL) is a flexible framework for learning a decision rule in an unknown environment. However, a large number of samples are often required for finding a useful decision rule. To mitigate this problem, the concept of transfer learning has been employed to utilize knowledge obtained from similar RL tasks. However, most approaches developed so far are useful only in low-dimensional settings. In this paper, we propose a novel transfer learning idea that targets problems with high-dimensional states. Our idea is to transfer knowledge between state factors (e.g., interacting objects) within a single RL task. This allows the agent to learn the system dynamics of the target RL task with fewer data samples. The effectiveness of the proposed method is demonstrated through experiments.

Human activity recognition from accelerometer data (e.g., obtained by smart phones) is gathering a great deal of attention since it can be used for various purposes such as remote health-care. However, since collecting labeled data is bothersome for new users, it is desirable to utilize data obtained from existing users. In this paper, we formulate this adaptation problem as learning under covariate shift, and propose a cornputationally efficient probabilistic classification method based on adaptive importance sampling. The usefulness of the proposed method is demonstrated in real-world human activity recognition. (C) 2011 Elsevier B.V. All rights reserved.

Policy gradient is a useful model-free reinforcement learning approach, but it tends to suffer from instability of gradient estimates. In this paper, we analyze and improve the stability of policy gradient methods. We first prove that the variance of gradient estimates in the PGPE (policy gradients with parameter-based exploration) method is smaller than that of the classical REINFORCE method under a mild assumption. We then derive the optimal baseline for PGPE, which contributes to further reducing the variance. We also theoretically show that PGPE with the optimal baseline is more preferable than REINFORCE with the optimal baseline in terms of the variance of gradient estimates. Finally, we demonstrate the usefulness of the improved PGPE method through experiments. (c) 2011 Elsevier Ltd. All rights reserved.

Multi-label classification allows a sample to belong to multiple classes simultaneously, which is often the case in real-world applications such as audio tagging, image annotation, video search, and text mining. In such a multi-label scenario, taking into account correlation between multiple labels can boost the classification accuracy. However, this in turn makes classifier training more challenging because handling multiple labels tends to induce a high-dimensional optimization problem. In this paper, we propose a highly scalable multilabel classifier based on a computationally efficient classification algorithm called the least-squares probabilistic classifier. Through experiments, we show the usefulness of our proposed method.

Direct policy search is a promising reinforcement learning framework, in particular for controlling continuous, high-dimensional systems. Policy search often requires a large number of samples for obtaining a stable policy update estimator, and this is prohibitive when the sampling cost is expensive. In this letter, we extend an expectation-maximization-based policy search method so that previously collected samples can be efficiently reused. The usefulness of the proposed method, reward-weighted regression with sample reuse (R-3), is demonstrated through robot learning experiments. (This letter is an extended version of our earlier conference paper: Hachiya, Peters, & Sugiyama, 2009.)

Information-maximization clustering learns a probabilistic classifier in an unsupervised manner so that mutual information between feature vectors and cluster assignments is maximized. A notable advantage of this approach is that it only involves continuous optimization of model parameters, which is substantially easier to solve than discrete optimization of cluster assignments. However, existing methods still involve non-convex optimization problems, and therefore finding a good local optimal solution is not straightforward in practice. In this paper, we propose an alternative information-maximization clustering method based on a squared-loss variant of mutual information. This novel approach gives a clustering solution analytically in a computationally efficient way via kernel eigenvalue decomposition. Furthermore, we provide a practical model selection procedure that allows us to objectively optimize tuning parameters included in the kernel function. Through experiments, we demonstrate the usefulness of the proposed approach. Copyright 2011 by the author(s)/owner(s).

Least-squares policy iteration is a useful reinforcement learning method in robotics due to its computational e?ciency. However, it tends to be sensitive to outliers in observed rewards. In this paper, we propose an alternative method that employs the absolute loss for enhancing robustness and reliability. The proposed method is formulated as a linear programming problem which can be solved eficiently by standard optimization software, so the computational advantage is not sacrificed for gaining robustness and reliability. We demonstrate the usefulness of the proposed approach through a simulated robot-control task. Copyright © 2010 The Institute of Electronics, Information and Communication Engineers.

Least-squares policy iteration is a useful reinforcement learning method in robotics due to its computational efficiency. However, it tends to be sensitive to outliers in observed rewards. In this paper, we propose an alternative method that employs the absolute loss for enhancing robustness and reliability. The proposed method is formulated as a linear programming problem which can be solved efficiently by standard optimization software, so the computational advantage is not sacrificed for gaining robustness and reliability. We demonstrate the usefulness of the proposed approach through a simulated robot-control task.

Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. The effectiveness of the proposed method, which we call active policy iteration (API), is demonstrated through simulations with a batting robot. (C) 2010 Elsevier Ltd. All rights reserved.

Most conventional Reinforcement Learning (RL) algorithms aim to optimize decision-making rules in terms of the expected re-turns. However, especially for risk management purposes, other risk-sensitive criteria such as the value-at-risk or the expected shortfall are sometimes preferred in real applications. Here, we describe a parametric method for estimating density of the returns, which allows us to handle various criteria in a unified manner. We first extend the Bellman equation for the conditional expected return to cover a conditional probability density of the returns. Then we derive an extension of the TD-learning algorithm for estimating the return densities in an unknown environment. As test instances, several parametric density estimation algorithms are presented for the Gaussian, Laplace, and skewed Laplace distributions. We show that these algorithms lead to risk-sensitive as well as robust RL paradigms through numerical experiments.

Estimating the conditional mean of an input-output relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation that is suitable for multi-dimensional continuous variables. The basic idea of the proposed method is to express the conditional density in terms of the density ratio and the ratio is directly estimated without going through density estimation. Experiments using benchmark and robot transition datasets illustrate the usefulness of the proposed approach. Copyright © 2010 The Institute of Electronics, Information and Communication Engineers.

Standard Reinforcement Learning (RL) aims to optimize decision-making rules in terms of the expected return. However, especially for risk-management purposes, other criteria such as the expected shortfall are some-times preferred. Here, we describe a method of approximating the distribution of returns, which allows us to derive various kinds of information about the returns. We first show that the Bellman equation, which is a recursive formula for the expected return, can be extended to the cumulative return distribution. Then we derive a nonparametric return distribution estimator with particle smooth ing based on this extended Bellman equation. A key aspect of the proposed algorithm is to represent the recursion relation in the extended Bellman equation by a simple replacement procedure of particles associated with a state by using those of the successor state. We show that our algorithm leads to a risk-sensitive R.L paradigm. The usefulness of the proposed approach is demonstrated through numerical experiments. Copyright 2010 by the author(s)/owner(s).

Estimating the conditional mean of an inputoutput relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation. Our basic idea is to express the conditional density in terms of the ratio of unconditional densities, and the ratio is directly estimated without going through density estimation. Experiments using benchmark and robot transition datasets illustrate the usefulness of the proposed approach. Copyright 2010 by the authors.

Model-free reinforcement learning (RL) is a machine learning approach to decision making in unknown environments. However, real-world RL tasks often involve high-dimensional state spaces, and then standard RL methods do not perform well. In this paper, we propose a new feature selection framework for coping with high dimensionality. Our proposed framework adopts conditional mutual information between return and state-feature sequences as a feature selection criterion, allowing the evaluation of implicit state-reward dependency. The conditional mutual information is approximated by a least-squares method, which results in a computationally efficient feature selection procedure. The usefulness of the proposed method is demonstrated on grid-world navigation problems.

Off-policy reinforcement learning is aimed at efficiently using data samples gathered from a policy that is different from the currently optimized policy. A common approach is to use importance sampling techniques for compensating for the bias of value function estimators caused by the difference between the data-sampling policy and the target policy. However, existing off-policy methods often do not take the variance of the value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations. © 2009 Elsevier Ltd. All rights reserved.

Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. We demonstrate the usefulness of the proposed method, named active policy iteration (API), through simulations with a batting robot.

Off-policy reinforcement learning is aimed at efficiently using data samples gathered from a policy that is different from the currently optimized policy. A common approach is to use importance sampling techniques for compensating for the bias of value function estimators caused by the difference between the data-sampling policy and the target policy. However, existing off-policy methods often do not take the variance of the value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. The usefulness of the proposed approach is demonstrated through simulated swing-up inverted-pendulum problem.

Direct policy search is a, promising reinforcement learning framework in particular for controlling in continuous, high-dimensional systems such as anthropomorphic robots. Policy search often requires a large number of samples for obtaining a stable policy update estimator due to its high flexibility. However, this is prohibitive when the sampling cost is expensive. Ill this paper, we extend all EM-based policy search method so that previously collected samples call be efficiently reused. The usefulness of the proposed method. called Reward-weighted Regression with. sample Reuse (R-3), is demonstrated through a robot learning experiment.

Least-squares policy iteration is a useful reinforcement learning method in robotics due to its computational efficiency. However, it tends to be sensitive to outliers in observed rewards. In this paper, we propose an alternative method that employs the absolute loss for enhancing robustness and reliability. The proposed method is formulated as a linear programming problem which can be solved efficiently by standard optimization software, so the computational advantage is not sacrificed for gaining robustness and reliability. We demonstrate the usefulness of the proposed approach through simulated robot-control tasks.

The least-squares policy iteration approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in simulated robot arm control and Khepera robot navigation.

Off-policy reinforcement learning is aimed at efficiently reusing data samples gathered in the past, which is an essential problem for physically grounded AI as experiments are usually prohibitively expensive. A common approach is to use importance sampling techniques for compensating for the bias caused by the difference between data-sampling policies and the target policy. However, existing off-policy methods do not often take the variance of value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a variant of cross-validation. We demonstrate the usefulness of the proposed approach through simulations. Copyright © 2008.

The least squares approach works efficiently in value function approximation, given appropriate basis functions. Because of its smoothness, the Gaussian kernel is a popular and useful choice as a basis function. However, it does not allow for discontinuity which typically arises in real-world reinforcement learning tasks. In this paper, we propose a new basis function based on geodesic Gaussian kernels, which exploits the non-linear manifold structure induced by the Markov decision processes. The usefulness of the proposed method is successfully demonstrated in a simulated robot arm control and Khepera robot navigation.

In this paper, we propose a novel method ”laserVAE” for learning feature descriptors of scan data from a 2D LIDAR, which are suitable for self-localization of a mobile robot in an environment. Our laserVAE is an enhanced version of variational autoencoder, which is tuned up for managing step-edges in scan data. Through experiments in a real environment, we demonstrate the effectiveness of the proposed method.

In this paper, we propose a novel method ”laserVAE” for learning feature descriptors of scan data from a 2D LIDAR, which are suitable for self-localization of a mobile robot in an environment. Our laserVAE is an enhanced version of variational autoencoder, which is tuned up for managing step-edges in scan data. Through experiments in a real environment, we demonstrate the effectiveness of the proposed method.

屋外環境でロボットをロバストに自律走行させるための、経路教示と経路計画を状況により切り替えるハイブリッド自律走行方法を提案

単眼カメラ画像から特定物体の距離を推定するディープラーニング方法の提案とロボットナビゲーションへの応用

単眼カメラ画像から特定物体の距離を推定するディープラーニング方法の提案

つくばチャレンジ2017にて、1kmの自律走行（マイルストーン2）を達成したロボットナビゲーション技術の解説

つくばチャレンジ2017の特定人物探索を題材に開発したディープラーニングを用いた単眼カメラ画像からの距離推定技術の解説

Locality sensitive hashing (LSH) is a computationally efficient alternative to the distance based anomaly detection. The main advantages of LSH lie in constant detection time, low memory requirement, and simple implementation. However, since the metric of distance in LSHs does not consider the property of normal training data, a naive use of existing LSHs would not perform well. In this paper, we propose a new hashing scheme so that hash functions are selected dependently on the properties of the normal training data for reliable anomaly detection. The distance metric of the proposed method, called NSH (Normality Sensitive Hashing) is theoretically interpreted in terms of the region of normal training data and its effectiveness is demonstrated through experiments on real-world data. Our results are favorably comparable to state-of-the arts with the low-level features.

Feature selection is a technique to screen out less important features. Many existing supervised feature selection algorithms use redundancy and relevancy as the main criteria to select features. However, feature interaction, potentially a key characteristic in real-world problems, has not received much attention. As an attempt to take feature interaction into account, we propose l_1-LSMI, an l_1-regularization based algorithm that maximizes a squared-loss variant of mutual information between selected features and outputs. Numerical results show that l_1-LSMI performs well in handling redundancy, detecting non-linear dependency, and considering feature interaction.

The policy gradient approach is a flexible and powerful reinforcement learning method particularly for problems with continuous actions such as robot control. A common challenge in this scenario is how to stabilize policy gradient estimates for reliable policy updates. In this paper, we combine the following three ideas and give a highly stable and practical policy gradient method: (a) the policy gradients with parameter based exploration, which is a recently proposed policy search method with high stability, (b) an importance sampling technique, which allows us to reuse previously gathered data in an unbiased way, and (c) an optimal baseline, which minimizes the variance of gradient estimates with their unbiasedness being maintained. For the proposed method, we give theoretical analysis of the variance of gradient estimates and show its usefulness through experiments.

Multi-label classification allows a sample to belong to multiple classes simultaneously, which is often the case in real-world applications such as audio tagging, image annotation, video search, and text mining. In such a multi-label scenario, taking into account correlation between multiple labels can boost the classification accuracy. However, this in turn makes classifier training more challenging because handling multiple labels tends to induce a high-dimensional optimization problem. In this paper, we propose a highly scalable multi-label classifier based on a computationally efficient classification algorithm called the least-squares probabilistic classifier. Through experiments, we show the usefulness of our proposed method.

Divergence estimators based on direct approximation of density-ratios without going through separate approximation of numerator and denominator densities have been successfully applied to machine learning tasks that involve distribution comparison such as outlier detection, transfer learning, and two-sample homogeneity test. However, since density-ratio functions often possess high fluctuation, divergence estimation is still a challenging task in practice. In this paper, we propose to use relative divergences for distribution comparison, which involves approximation of relative density-ratios. Since relative density-ratios are always smoother than corresponding ordinary density-ratios, our proposed method is favorable in terms of the non-parametric convergence speed. Furthermore, we show that the proposed divergence estimator has asymptotic variance independent of the model complexity under a parametric setup, implying that the proposed estimator hardly overfits even with complex models. Through experiments, we demonstrate the usefulness of the proposed approach.

The natural policy gradient method was shown to be a useful approach to policy search in reinforcement learning. However, its potential weakness is that information on returns is not reflected in the metric of natural gradients, implying that it is not adaptive to data and thus less flexible. To overcome this, we propose to use Newton's method which uses the Hessian of the expected return as a metric. However, the naive implementation of Newton's method does not guarantee the Hessian to be negative definite, which causes instability on policy updates. To cope with this problem, we propose an adaptive scheme to keep the Hessian nonnegative. We demonstrate the effectiveness of our proposed method in standard reinforcement learning tasks.

Oriental ink painting, called Sumi-e, is one of the most appealing painting styles that has attracted artists around the world. The major challenges in computer-based Sumi-e simulation are to abstract complex scene information and draw smooth and natural brush strokes. To automatically find such strokes, we propose to model the brush as a reinforcement-learning (RL) agent, and learn desired brush-trajectories by maximizing the sum of rewards in the policy search framework. We also provide elaborate design of state space, action space, and a reward function tailored for a Sumi-e agent. The effectiveness of our proposed approach is demonstrated through simulated Sumi-e experiments.

Oriental ink painting, called Sumi-e, is one of the most appealing painting styles that has attracted artists around the world. The major challenges in computer-based Sumi-e simulation are to abstract complex scene information and draw smooth and natural brush strokes. To automatically find such strokes, we propose to model the brush as a reinforcement-learning (RL) agent, and learn desired brush-trajectories by maximizing the sum of rewards in the policy search framework. We also provide elaborate design of state space, action space, and a reward function tailored for a Sumi-e agent. The effectiveness of our proposed approach is demonstrated through simulated Sumi-e experiments.

Oriental ink painting, called Sumi-e, is one of the most appealing painting styles that has attracted artists around the world. The major challenges in computer-based Sumi-e simulation are to abstract complex scene information and draw smooth and natural brush strokes. To automatically find such strokes, we propose to model the brush as a reinforcement-learning (RL) agent, and learn desired brush-trajectories by maximizing the sum of rewards in the policy search framework. We also provide elaborate design of state space, action space, and a reward function tailored for a Sumi-e agent. The effectiveness of our proposed approach is demonstrated through simulated Sumi-e experiments.Oriental ink painting, called Sumi-e, is one of the most appealing painting styles that has attracted artists around the world. The major challenges in computer-based Sumi-e simulation are to abstract complex scene information and draw smooth and natural brush strokes. To automatically find such strokes, we propose to model the brush as a reinforcement-learning (RL) agent, and learn desired brush-trajectories by maximizing the sum of rewards in the policy search framework. We also provide elaborate design of state space, action space, and a reward function tailored for a Sumi-e agent. The effectiveness of our proposed approach is demonstrated through simulated Sumi-e experiments.

A recently-proposed information-maximization clustering method (Gomes et al., NIPS2010) learns a kernel logistic regression classifier in an unsupervised manner so that mutual information between feature vectors and cluster assignments is maximized. A notable advantage of this approach is that it only involves continuous optimization of a logistic model, which is substantially easier than discrete optimization of cluster assignments. However, this method still suffers from two weaknesses: (i) manual tuning of kernel parameters is necessary, and (ii) finding a good local optimal solution is not straightforward due to the strong non-convexity of logistic-regression learning. In this paper, we first show that the kernel parameters can be systematically optimized by maximizing mutual information estimates. We then propose an alternative information-maximization clustering approach using a squared-loss variant of mutual information. This novel approach allows us to obtain clustering solutions analytically in a computationally very efficient way. Through experiments, we demonstrate the usefulness of the proposed approaches.

標準的な強化学習の枠組みでは, Bellman方程式を用いてリターン(割引報酬和)の期待値を推定し,意思決定を行う.近年,我々はこの枠組みを拡張し,分布Bellman方程式を用いてリターンの分布を推定する方法を提案した.これにより,バリュー・アット・リスクなどのリスクを考慮した任意の指標に基づく意思決定が行えるようになったが,分布Bellman方程式を用いた手法の収束性などの理論的性質は末だ解明されていない.本論文では,分布Bellman方程式を動的計画法によって解いた場合,解が初期近似分布に依存せず常に真のリターン分布に収束することを証明する.さらに,リターン分布推定量のモーメントの収束率も示す.最後に,得られた理論結果から,既存のリターン分布推定法の改善方法を提案し,数値実験からその有効性を示す.

Model-free reinforcement learning (RL) is a machine learning approach to decision making in unknown environment. However, real-world RL tasks often involve high-dimensional state space, and then standard RL methods do not perform well. In this paper, we propose a new feature selection framework for coping with high dimensionality. Our proposed framework adopts conditional mutual information between state and return sequences as a feature selection criterion, allowing the evaluation of implicit state-reward dependency. The conditional mutual information is approximated by a least-squares method, which results in a computationally efficient feature selection procedure. The usefulness of the proposed method is demonstrated on simulated mobile-robot navigation experiments.

We introduce an extension to standard reinforcement learning setting called observational RL (ORL) where additional observational information is available to the agent. This allows the agent to learn the system dynamics with fewer data samples, which is an essential feature for practical applications of RL methods. We show that ORL can be formulated as a multitask learning problem. A similarity-based and a component-based multitask learning methods are proposed for learning the transition probabilities of the ORL problem. The effectiveness of the proposed methods is evaluated in experiments of grid world and object lifting tasks.We introduce an extension to standard reinforcement learning setting called observational RL (ORL) where additional observational information is available to the agent. This allows the agent to learn the system dynamics with fewer data samples, which is an essential feature for practical applications of RL methods. We show that ORL can be formulated as a multitask learning problem. A similarity-based and a component-based multitask learning methods are proposed for learning the transition probabilities of the ORL problem. The effectiveness of the proposed methods is evaluated in experiments of grid world and object lifting tasks.

Estimating the conditional mean of an input-output relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation that is suitable for multi-dimensional continuous variables. The basic idea of the proposed method is to express the conditional density in terms of the density ratio and the ratio is directly estimated without going through density estimation.Estimating the conditional mean of an input-output relation is the goal of regression. However, regression analysis is not sufficiently informative if the conditional distribution has multi-modality, is highly asymmetric, or contains heteroscedastic noise. In such scenarios, estimating the conditional distribution itself would be more useful. In this paper, we propose a novel method of conditional density estimation that is suitable for multi-dimensional continuous variables. The basic idea of the proposed method is to express the conditional density in terms of the density ratio and the ratio is directly estimated without going through density estimation.

Appropriately designing sampling policies is highly important for obtaining better control policies in reinforcement learning. In this paper, we first show that the least-squares policy iteration (LSPI) framework allows us to employ statistical active learning methods for linear regression. Then we propose a design method of good sampling policies for efficient exploration, which is particularly useful when the sampling cost of immediate rewards is high. The proposed method combined with LSPI is called active policy iteration (API). Through simulations we demonstrate the usefulness of API.

Direct policy search is a promising reinforcement learning framework in particular for controlling in continuous, high-dimensional systems such as anthropomorphic robot. Policy search often requires a large number of samples for obtaining a stable policy update estimator due to its high flexibility. However, this is prohibitive when the sampling cost is expensive. In this paper, we extend an EM-based policy search method so that previously collected samples can be efficiently reused. The usefulness of the proposed method, called Reward-weighted Regression with sample Reuse (R^3), is demonstrated through a toy example.

Off-policy reinforcement learning is aimed at efficiently reusing data samples gathered in the past. A common approach is to use importance sampling techniques for compensating for the bias caused by the difference between data-collecting policies and the target policy. However, existing off-policy methods do not often take the variance of value function estimators explicitly into account and therefore their performance tends to be unstable. To cope with this problem, we propose using an adaptive importance sampling technique which allows us to actively control the trade-off between bias and variance. We further provide a method for optimally determining the trade-off parameter based on a statistical machine learning theory.

助言・指導

機械学習コンサルティングの助言及び指導

客員研究員

各種センサーデータの解析及び防災・減災シミュレータの研究開発

学術雑誌等の編集委員・査読・審査員等

Reviewer

講演講師等

講師,任期:2019年11月～

公開講座・講演会の企画・講師等

機械学習の概要、応用例およびPythonを用いた機械学習の実装演習,日付:2019.5.29

講演講師等

講師,任期:2019年5月～

小・中・高校生を対象とした学部体験入学・出張講座等

和歌山県立向陽高校にて、機械学習入門の出張講義を実施,日付:2019年11月8日

公開講座・講演会の企画・講師等

最先端の機械学習とその応用,日付:2019.3.5

講演講師等

講師,任期:2019年3月～

公開講座・講演会の企画・講師等

ディープラーニングを用いたセンサ ーデータの圧縮と変換,日付:2018.12.13

学術雑誌等の編集委員・査読・審査員等

Reviewer

学術雑誌等の編集委員・査読・審査員等

Reviewer

学術雑誌等の編集委員・査読・審査員等

Program Committee

公開講座・講演会の企画・講師等

機械学習の研究に関する講演およびポスター発表,日付:2017.11.10

学術雑誌等の編集委員・査読・審査員等

Reviewer

学術雑誌等の編集委員・査読・審査員等

Program Committee

公開講座・講演会の企画・講師等

機械学習の研究に関する講演,日付:2017.8.10

学術雑誌等の編集委員・査読・審査員等

Reviewer

学術雑誌等の編集委員・査読・審査員等

Associate Editor

国や地方自治体、他大学・研究機関等での委員

客員研究員,任期:2018年4月～2019年3月

学協会、政府、自治体等の公的委員

防災科学チームにおいて、各種センサーデータの解析および 防災・減災シミュレータの研究開発

学協会、政府、自治体等の公的委員

ロボカップジュニアポスター審査

学協会、政府、自治体等の公的委員

防災科学チームにおいて、各種センサーデータの解析および 防災・減災シミュレータの研究開発