ML-Agentsでの学習時にエージェントが動かない

Question

実現したいこと

UnityのML-Agentsを使って強化学習をしたいと考えています。
フィールド上のエージェントが落下やランダムな位置と角度で配置された障害物との接触を避けながらゴールを目指すものです。

発生している問題

学習を開始してもエージェントが全く動かず、学習が進みません。
手動でWASDキーで動かすことはできるのですが、学習モードの時には動きません。
おそらくエージェントを動かす部分のコード(下記のMoveAgentの部分)が誤っていると思うのですが、どのように誤っているのかわかりません。
同じ環境で別の学習をした際には上手くいったので、環境が原因ではないと思われます。
ご教授よろしくお願い致します。

ソースコード

using System.Collections;
using System.Collections.Generic;
using System.Linq;
using UnityEngine;
using Unity.MLAgents;
using Unity.MLAgents.Sensors;
using Unity.MLAgents.Actuators;

public class FWAgent : Agent
{
    Rigidbody rBody;
    public Transform Target;
    public GameObject RhwystrPlace;

    void Start()
    {
        rBody = GetComponent<Rigidbody>();
    }

    public override void OnEpisodeBegin()
    {
        this.rBody.angularVelocity = Vector3.zero;
        this.rBody.velocity = Vector3.zero;
        this.transform.localPosition = new Vector3(5.2f, 0.5f, -5.2f);
        this.transform.localRotation = Quaternion.identity;

        GameObject[] objects;
        objects = GameObject.FindGameObjectsWithTag("Rhwystr");
        for(int d = 0; d < objects.Length; d++)
        {
            Destroy(objects[d].gameObject);
        }
        RhwystrPlace.GetComponent<RhwyGenerater>().PutRhwy();
    }

    public override void CollectObservations(VectorSensor sensor)
    {
        sensor.AddObservation(Target.localPosition);
        sensor.AddObservation(this.transform.localPosition);

        sensor.AddObservation(rBody.velocity.x);
        sensor.AddObservation(rBody.velocity.z);
    }

    public void MoveAgent(ActionSegment<int> act)
    {
        var dirToGo = Vector3.zero;
        var rotateDir = Vector3.zero;

        var action = act[0];
        switch (action)
        {
            case 1:
                dirToGo = transform.position += transform.forward * 1.0f * Time.deltaTime;
                break;
            case 2:
                dirToGo = transform.position -= transform.forward * 1.0f * Time.deltaTime;
                break;
            case 3:
                rotateDir = transform.up * 0.5f;
                break;
            case 4:
                rotateDir = transform.up * -0.5f;
                break;
        }
        transform.Rotate(rotateDir, Time.deltaTime * 100f);
    }

    void OnCollisionEnter(Collision collision)
    {
        if (collision.gameObject.name == "Target")
        {
            AddReward(1.0f);
            EndEpisode();
        }
        else
        {
            AddReward(-0.2f);
            EndEpisode();
        }
    }

    private void FixedUpdate()
    {
        AddReward(-0.001f);
    }

    public override void OnActionReceived(ActionBuffers actionBuffers)
    {
        MoveAgent(actionBuffers.DiscreteActions);

        Vector3 AgentPos = this.transform.localPosition;
        Vector3 TargetPos = Target.localPosition;
        float AgentFall = this.transform.localPosition.y;
        float distanceToTarget = Vector3.Distance(AgentPos, TargetPos);

        if (AgentFall < 0.45)
        {
            AddReward(-0.2f);
            EndEpisode();
        }
    }

    public override void Heuristic(in ActionBuffers actionsOut)
    {
        var discreteActionsOut = actionsOut.DiscreteActions;
        discreteActionsOut[0] = 0;
        if (Input.GetKey(KeyCode.W))
        {
            discreteActionsOut[0] = 1;
        }
        else if (Input.GetKey(KeyCode.S))
        {
            discreteActionsOut[0] = 2;
        }
        else if (Input.GetKey(KeyCode.D))
        {
            discreteActionsOut[0] = 3;
        }
        else if (Input.GetKey(KeyCode.A))
        {
            discreteActionsOut[0] = 4;
        }
    }
}

######YAMLファイル

behaviors:
  FWAgent:
    trainer_type: ppo
    hyperparameters:
      batch_size: 128
      buffer_size: 2048
      learning_rate: 3.0e-4
      beta: 5.0e-4
      epsilon: 0.2
      lambd: 0.95
      num_epoch: 3
      learning_rate_schedule: linear
    network_settings:
      normalize: false
      hidden_units: 128
      num_layers: 2
    reward_signals:
      extrinsic:
        gamma: 0.99
        strength: 1.0
    max_steps: 500000
    time_horizon: 128
    summary_freq: 10000

###ターミナルでの出力

2021-07-12 00:13:30 INFO [stats.py:139] FWAgent. Step: 10000. Time Elapsed: 47.074 s. Mean Reward: -0.200. Std of Reward: 0.000. Training.
2021-07-12 00:14:12 INFO [stats.py:139] FWAgent. Step: 20000. Time Elapsed: 89.638 s. No episode was completed since last summary. Training.
2021-07-12 00:14:58 INFO [stats.py:139] FWAgent. Step: 30000. Time Elapsed: 135.142 s. No episode was completed since last summary. Training.
2021-07-12 00:15:46 INFO [stats.py:139] FWAgent. Step: 40000. Time Elapsed: 183.530 s. No episode was completed since last summary. Training.
2021-07-12 00:16:35 INFO [stats.py:139] FWAgent. Step: 50000. Time Elapsed: 231.954 s. No episode was completed since last summary. Training.
2021-07-12 00:17:24 INFO [stats.py:139] FWAgent. Step: 60000. Time Elapsed: 281.103 s. No episode was completed since last summary. Training.
2021-07-12 00:18:10 INFO [stats.py:139] FWAgent. Step: 70000. Time Elapsed: 327.051 s. No episode was completed since last summary. Training.
2021-07-12 00:18:52 INFO [stats.py:139] FWAgent. Step: 80000. Time Elapsed: 369.530 s. No episode was completed since last summary. Training.
2021-07-12 00:19:35 INFO [stats.py:139] FWAgent. Step: 90000. Time Elapsed: 412.841 s. No episode was completed since last summary. Training.
2021-07-12 00:20:25 INFO [stats.py:139] FWAgent. Step: 100000. Time Elapsed: 462.921 s. No episode was completed since last summary. Training.

###Unityの画面

######学習中の画面

環境

OS: macOS 10.14.6
Unity: Version 2020.3.13f1 Personal
ML-Agents: Release12 (Python Package 0.23.0)
Python: 3.7.6

Answer 1

エージェントのコードを拝見しますに、取りうる行動の選択肢は...

• 0: 何もしない
• 1: 前進する
• 2: 後退する
• 3: 右に回る
• 4: 左に回る

の5パターンのように見えます。ですがスクリーンショットではBehaviorParametersの「Actions」→「Discrete Branches」→「Branch 0 Size」が1となっており、おそらく常に「0: 何もしない」が選択されるために動かないんじゃないでしょうか?
試しに「Branch 0 Size」を5にしてみるとどうでしょうか。