常連

自己紹介

### 前提・実現したいこと

python3のGoogle Colaboratoryでfig.canvas.draw()が実行されない

[https://nbviewer.jupyter.org/urls/bitbucket.org/yairmau/notebooks/raw/master/tdgle.ipynb](https://nbviewer.jupyter.org/urls/bitbucket.org/yairmau/notebooks/raw/master/tdgle.ipynb) で公開されている方程式のシミュレーションを行いたいのですが、最後のセルだけがエラーこそ無いものの何も表示されずに困っております。（fig.canvas.draw()が実行されない？）


([https://www.youtube.com/watch?v=JgE9Px7zsQE](https://www.youtube.com/watch?v=JgE9Px7zsQE) のように時間tごとに連続的に表示させたいです)

ノートブック上ではなく**PC上では問題なく実行できる**のですが、ノートブック環境でも実行できるようにしたいです。
### 発生している問題・エラーメッセージ


下に記載しているGoogle Colaboratory上のコードでセル[1]~[4]までは正常に実行できていそうだが、セル[5]の部分のみ実行しても何も表示されない


### 該当のソースコード

```python3
#Google Colaboratory上で実行しています

#[1]
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.axes_grid1 import make_axes_locatable
import time as tm


#[2]
# grid size
n = 128
# for 1d simulation write N=(n,)
N=(n,n)
# diffusion coefficient
D = 1.0
# spatial dimensions
L = 100.0
dx = L / n
x = np.arange(0,L,dx)
# time
t = 0.0
total_time = 3.0
# beware of the Von Neumann stability analysis
# https://en.wikipedia.org/wiki/Von_Neumann_stability_analysis
dt = 0.2 * 0.5 * dx**2 / D


#[3]
# define functions
def periodic_lap_1d(u,dx=1.0):
    return (+1*np.roll(u,+1)
            +1*np.roll(u,-1)
            -2*u) / dx**2
def periodic_lap_2d(u,dx=1.0):
    return (+1*np.roll(u,+1,axis=0)
            +1*np.roll(u,-1,axis=0)
            +1*np.roll(u,+1,axis=1)
            +1*np.roll(u,-1,axis=1)
            -4*u) / dx**2
f = lambda u: u - u**3


#[4]
# initialize and start plotting
plt.ion()
fig = plt.figure(1,figsize=(7,6))
plt.clf()
ax = fig.add_subplot(111)
# random initial condition
u = 2*np.random.random(N)-1.0
if len(N) == 1:
    lap = periodic_lap_1d
    p, = ax.plot(x,u)
    ax.axis([x[0],x[-1],-1.1,1.1])
if len(N) == 2:
    lap = periodic_lap_2d
    p = ax.imshow(u,cmap="RdGy", vmin=-1.0, vmax=1.0,extent=[0,L,0,L])
    # create an axes on the right side of ax. The width of cax will be 5%
    # of ax and the padding between cax and ax will be fixed at 0.15 inch.
    divider = make_axes_locatable(ax)
    colorbar_ax = divider.append_axes("right", size="5%", pad=0.15)
    cbar = fig.colorbar(p, cax=colorbar_ax, ticks=[-1,-0.5,0,0.5,1])
ax.set_title("time={:5.1f}".format(0.0))
plt.show()


#[5] このセルが正常に実行できていない？
# start simulation
while t<total_time: # t=0.0, total_time=3.0
    t += dt         # dt≒0.061
    u = u + dt * (f(u) + D * lap(u,dx) )
    # we don't need to plot again, just to update the data of the plot
    if len(N) == 1:
        p.set_data(x,u)
    if len(N) == 2:
        p.set_data(u)
    ax.set_title("time={:5.1f}".format(t))
    fig.canvas.draw()

```

### 試したこと

jupyter notebookでも上のコードを試したのですが同様にセル[5](画像上ではセル[6])のみエラーは起こらないものの何も表示されませんでした
また、plt.show()などを最後に挿入しても同様の結果でした


画像はGoogle Colaboratory上で実行した結果のキャプチャ画像です
![イメージ説明](59bbb607c65d7ce09ea4299844e55ba2.jpeg)
![イメージ説明](6784761cf12c0fed93313e8bc535f169.jpeg)
##### 補足情報
・OS,ソフトウェアのバージョン(コマンドプロンプト上のpip freezeより)
OS:Windows10
Python 3.7.3
numpy==1.16.2
matplotlib==3.0.3

・Google Colaboratoryの設定でGPUを使うように設定済み

・このpythonコードを公開されている先生のページです
[http://yairmau.com/tutorials/python/](http://yairmau.com/tutorials/python/)

[Python3]fig.canvas.draw()をGoogle Colaboratory上で実行したい

ネットワークスパラメーターが同じ状態での報酬のばらつきが大き過ぎます。
ネットパラメータはきちんとリストアできているのに、上記の問題が発生するということはコードに何らかのバグが存在するということになりますが、自分が確認してもどこにバグがあるのかわかりません。

コードは文字制限により一部とはなりますが、よろしくお願いします。

報酬の決定
```python
import numpy as np

def reward2(double trend,list pip,int action,int position,list states,double pip_cost,double spread):
    cdef list r = []
    cdef int sub = 0
    if position != 3:
        if action == 0:
            p = [(trend - s) * pip_cost for s in states]
        else:
            p = [(s - trend) * pip_cost for s in states]
            spread *= -1

        if action == position:
            sub = 0
            for b in range(0, len(p)):
                r = [-40.0, True] if p[b] <= -40 else [p[b], False]
                if r[1]:
                    pip.append(r[0])
                    states.pop(b - sub)
                    sub += 1
            states.append(trend+ spread)
            position = action
        else:
            for b in p:
              b = -40.0 if b <= -40 else b
              pip.append(b)
            states = [trend+ spread]
            position = action

    else:
        states = [trend + spread] if action == 0 else [trend - spread]
        position = action

    return states,pip,position
```
アクションの決定
```python
def exploration(prediction,output_size,e = 0.2):
  prediction = prediction.astype("float64")
  
  if np.random.rand() < e:
    tau=2.0
    clip=(-1.0, 1.0)
    exp_values = np.exp(np.clip(prediction / tau, clip[0], clip[1]))
    probs = exp_values / np.sum(exp_values)
    action = np.random.choice(range(output_size), p=probs)
  else:
    action = np.argmax(prediction)
  
  return action
```

強化学習のトレーニング
```python
    def train(self, iterations, checkpoint, spread, pip_cost, n=4):
        copy = 0
        for i in range(iterations):
            position = 3
            total_pip = 0.00
            pip = []
            spread = spread / pip_cost
            done = False
            states = []
            p = []
            h_s = []
            h_r = []
            h_i = []
            h_p = []
            self.init_value = np.zeros((1, 128))
            tau = 0
            old_reword = 0.0
            self.history = []
            self.best_pip = 0
            self.MEMORIES = deque()
            self.pip = []
            if (copy + 1) % 2 == 0: 
              self._assign('actor-original', 'actor-target')
              self._assign('critic-original', 'critic-target')
            for t in  range(0, len(self.trend)-1, self.skip):
                state = self.get_state(t)
                h_s.append(state)
                next_state = self.get_state(t + 1)
                h_i.append(self.init_value[0])
                action = self._select_action(state,next_state)
                self.history.append(action)
                h_p.append(self.pred)

                states,pip,position = self.reward(self.trend,t,pip,action,position,states,pip_cost,spread)

                if len(pip) != 0:
                  self.pip = np.asanyarray(pip)
                  total_pip = np.sum(self.pip)
                mean_pip = total_pip / (t + 1)
                reward = total_pip - old_reword + self.ri
                old_reword = total_pip
                h_r.append(reward)
                
                tau = t - n + 1
                if tau >= 0:
                  reward = self.discount_rewards([r for r in h_r[tau+1:tau+n]])
                  self._memorize(h_s[tau], self.history[tau], reward, next_state,h_p[tau], done, h_i[tau])
```

![イメージ説明](8cdc0c9ccdf06fc79cf6c30ca74cc6a7.jpeg)
![イメージ説明](dcf8a255d415502e6ab4ba6bc0844e31.jpeg)
![イメージ説明](6cb04c66bde14ec8ecbd8feb1ed7be66.jpeg)

コードを追加します。
```python
    def __init__(self, path, window_size, skip, save=False, saver_path=None, restore=False,sess=None, noise=True,norm=True):
        self.path = path
        self.window_size = window_size
        self._preproc()
        self.state_size = (None, self.window_size, self.df.shape[-1])
        self.skip = skip
        self.reward = reward
        self.memory = Memory(self.MEMORY_SIZE)
        self.mem = Memory
        #
        self.ent_coef = ent_coef
        self.target_entropy = target_entropy
        self.sess = sess
        self.actor = Actor('actor-original', self.state_size, self.OUTPUT_SIZE,noise,norm)
        self.critic = Critic('critic-original', self.state_size, self.OUTPUT_SIZE, self.LEARNING_RATE,noise,norm)
        self.critic_target = Critic('critic-target', self.state_size, self.OUTPUT_SIZE, self.LEARNING_RATE,noise,norm)
        self.icm = ICM(self.state_size, self.OUTPUT_SIZE, self.LEARNING_RATE)
        
        with tf.variable_scope("loss"):
          if self.target_entropy == 'auto':
            self.target_entropy = -np.prod(self.OUTPUT_SIZE).astype(np.float32)
          self.target_entropy = float(self.target_entropy)
        
          entropy = tf.reduce_mean(self.actor.entropy)
        
          if isinstance(self.ent_coef, str) and self.ent_coef.startswith('auto'):
            init_value = 1.0
            if '_' in self.ent_coef:
              init_value = float(self.ent_coef.split('_')[1])
              assert init_value > 0., "The initial value of ent_coef must be greater than 0"
            
            self.log_ent_coef = tf.get_variable('log_ent_coef', dtype=tf.float32,initializer=np.log(init_value).astype(np.float32))
            self.ent_coef = tf.exp(self.log_ent_coef)
          else:
            self.ent_coef = float(self.ent_coef)
        
          min_qf = tf.minimum(self.critic.qf1, self.critic.qf2)
          ent_coef_loss, entropy_optimizer = None, None
          if not isinstance(self.ent_coef, float):
            ent_coef_loss = -tf.reduce_mean(
                self.log_ent_coef * tf.stop_gradient(self.actor.log_pi + self.target_entropy))
        
          policy_kl_loss = tf.reduce_mean(self.ent_coef * self.actor.log_pi - self.critic.qf1)
          policy_loss = policy_kl_loss
          v_backup = tf.stop_gradient(min_qf - self.ent_coef * self.actor.log_pi)
          value_loss = 0.5 * tf.reduce_mean((self.critic.value_fn - v_backup) ** 2)
          value_loss += self.critic.td_loss
          self.policy_loss = policy_loss
          self.actor_optimizer = tf.train.AdamOptimizer(self.LEARNING_RATE,name="actor_optimizer").minimize(policy_loss,var_list=get_vars("actor-original"))
          self.vf_optimizer = tf.train.AdamOptimizer(self.LEARNING_RATE,name="actor_optimizer").minimize(value_loss,var_list=get_vars("critic-original"))
          self.entropy_optimizer = tf.train.AdamOptimizer(learning_rate=self.LEARNING_RATE,name="entropy_optimizer").minimize(ent_coef_loss,var_list=self.log_ent_coef)
        
        self.save = save
        self.saver = tf.train.Saver(tf.global_variables())
        self.actor_saver = tf.train.Saver(tf.get_collection(tf.GraphKeys.GLOBAL_VARIABLES, "actor-original"))
        self.saver_path = saver_path
          
        if restore==True:
          self.saver.restore(self.sess,self.saver_path)
          self.actor_saver.restore(self.sess,"actor_sac")
        else:
          self.sess.run(tf.global_variables_initializer())
```

環境の準備
```ここに言語を入力
    def _preproc(self):
        df = pd.read_csv(self.path)
        X = df[["Close"]]
        X = MinMaxScaler().fit_transform(X)
#         X = np.asanyarray(X)
        
        gen = tf.keras.preprocessing.sequence.TimeseriesGenerator(X, np.asanyarray(df[["Open"]])[-len(X)::], self.window_size)
        x = []
        y = []
        for i in gen:
            x.extend(i[0].tolist())
        x = np.asanyarray(x)

        self.df = x[-self.STEP_SIZE::]
        self.trend = np.asanyarray(df[["Open"]])[-len(self.df)::]
```

探査
```ここに言語を入力
    def _select_action(self, state,next_state=None):
        prediction, self.init_value = self.sess.run([self.actor.logits,self.actor.last_state],
                                                     feed_dict={self.actor.X:[state],self.actor.initial_state:self.init_value})
        self.pred = prediction
        action = exploration(prediction[0], self.OUTPUT_SIZE,self.EPSILON)
        if next_state is not None:
          self.ri = self.sess.run(self.icm.ri,
                                  feed_dict={self.icm.state:[state],self.icm.next_state:[next_state],self.icm.action:action.reshape((-1,1))})
        return action
```
エージェントの評価
```ここに言語を入力
    def buy(self, spread, pip_cost, sl):
        position = 3
        pip = []
        states = []
        spread = spread / pip_cost
        loscut = False
        self.init_value = np.zeros((1, 512))
        for t in  range(0, len(self.trend), self.skip):
                state = self.get_state(t)
                action = self._select_action(state)
              
                states,pip,position = self.reward(self.trend,t,pip,action,position,states,pip_cost,spread)

                if len(pip) != 0:
                  self.pip = np.asanyarray(pip)
                  total_pip = np.sum(self.pip)
              # st ate = next_state
        return total_pip, self.pip
```