StyleGAN2をGoogle Colaboratoryで実装したい

Question

前提・実現したいこと

Google ColaboratoryでStyleGAN2を実装しようとしています。(初学者です)
imagenet を用いて実装し、画像Aから画像Bへのトランジションを行いたいです。

主に以下のサイトを参考にして進めています。
・https://techpr.info/ml/stylegan2-colab/
・https://github.com/justinpinkney/awesome-pretrained-stylegan2

発生している問題・エラーメッセージ

先週まで実行できていたのに急に "RuntimeError: No GPU devices found" が出てGPUが使えなくなりました。
先週からコードは何も触っていません。

--2022-10-12 01:49:51--  https://battle.shawwn.com/sdc/stylegan2-imagenet-512/model.ckpt-533504.pkl
Resolving battle.shawwn.com (battle.shawwn.com)... 104.21.82.78, 172.67.155.55, 2606:4700:3036::6815:524e, ...
Connecting to battle.shawwn.com (battle.shawwn.com)|104.21.82.78|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 364035409 (347M) [application/octet-stream]
Saving to: ‘model.ckpt-533504.pkl.1’

model.ckpt-533504.p 100%[===================>] 347.17M  26.9MB/s    in 14s     

2022-10-12 01:50:06 (24.6 MB/s) - ‘model.ckpt-533504.pkl.1’ saved [364035409/364035409]

Local submit - run_dir: results/00001-generate-images
dnnlib: Running run_generator.generate_images() on localhost...
Loading networks from "model.ckpt-533504.pkl"...
2022-10-12 01:50:08.183797: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:08.183918: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:08.184002: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:08.184080: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:08.184159: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:08.184377: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:08.184527: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:08.184546: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1662] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Setting up TensorFlow plugin "fused_bias_act.cu": Preprocessing... 2022-10-12 01:50:09.054726: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:09.054855: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcublas.so.10.0'; dlerror: libcublas.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:09.054932: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcufft.so.10.0'; dlerror: libcufft.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:09.055003: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcurand.so.10.0'; dlerror: libcurand.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:09.055075: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusolver.so.10.0'; dlerror: libcusolver.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:09.055152: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcusparse.so.10.0'; dlerror: libcusparse.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:09.055228: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudnn.so.7'; dlerror: libcudnn.so.7: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia
2022-10-12 01:50:09.055246: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1662] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
Failed!
Traceback (most recent call last):
  File "/content/stylegan2/run_generator.py", line 168, in <module>
    main()
  File "/content/stylegan2/run_generator.py", line 163, in main
    dnnlib.submit_run(sc, func_name_map[subcmd], **kwargs)
  File "/content/stylegan2/dnnlib/submission/submit.py", line 343, in submit_run
    return farm.submit(submit_config, host_run_dir)
  File "/content/stylegan2/dnnlib/submission/internal/local.py", line 22, in submit
    return run_wrapper(submit_config)
  File "/content/stylegan2/dnnlib/submission/submit.py", line 280, in run_wrapper
    run_func_obj(**submit_config.run_func_kwargs)
  File "/content/stylegan2/run_generator.py", line 21, in generate_images
    _G, _D, Gs = pretrained_networks.load_networks(network_pkl)
  File "/content/stylegan2/pretrained_networks.py", line 76, in load_networks
    G, D, Gs = pickle.load(stream, encoding='latin1')
  File "/content/stylegan2/dnnlib/tflib/network.py", line 297, in __setstate__
    self._init_graph()
  File "/content/stylegan2/dnnlib/tflib/network.py", line 154, in _init_graph
    out_expr = self._build_func(*self.input_templates, **build_kwargs)
  File "<string>", line 495, in G_synthesis_stylegan2
  File "<string>", line 459, in layer
  File "<string>", line 103, in modulated_conv2d_layer
  File "<string>", line 72, in apply_bias_act
  File "/content/stylegan2/dnnlib/tflib/ops/fused_bias_act.py", line 68, in fused_bias_act
    return impl_dict[impl](x=x, b=b, axis=axis, act=act, alpha=alpha, gain=gain)
  File "/content/stylegan2/dnnlib/tflib/ops/fused_bias_act.py", line 122, in _fused_bias_act_cuda
    cuda_kernel = _get_plugin().fused_bias_act
  File "/content/stylegan2/dnnlib/tflib/ops/fused_bias_act.py", line 16, in _get_plugin
    return custom_ops.get_plugin(os.path.splitext(__file__)[0] + '.cu')
  File "/content/stylegan2/dnnlib/tflib/custom_ops.py", line 130, in get_plugin
    compile_opts += ' --gpu-architecture=%s' % _get_cuda_gpu_arch_string()
  File "/content/stylegan2/dnnlib/tflib/custom_ops.py", line 52, in _get_cuda_gpu_arch_string
    raise RuntimeError('No GPU devices found')
RuntimeError: No GPU devices found

該当のソースコード

!pip uninstall tensorflow -y
!pip install tensorflow-gpu==1.15
!pip install keras==2.2.4

!git clone https://github.com/NVlabs/stylegan2.git

!wget https://battle.shawwn.com/sdc/stylegan2-imagenet-512/model.ckpt-533504.pkl
# 画像生成の実行
!python /content/stylegan2/run_generator.py generate-images \
    --network=model.ckpt-533504.pkl \
  --seeds=6600-6625 --truncation-psi=0.5

試したこと

調べたら pip tensorflow-gpu をインストールするとありましたが変わりませんでした。
ランタイムの再起動なども試しました。

補足情報（FW/ツールのバージョンなど）

Wed Oct 12 02:58:04 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.03    Driver Version: 460.32.03    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Tesla T4            Off  | 00000000:00:04.0 Off |                    0 |
| N/A   38C    P8     9W /  70W |      0MiB / 15109MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Comments

[jbpb0]

> 2022-10-12 01:50:08.183797: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia

cuda 10.0が無くなってますね

もしかしたら、
https://qiita.com/katoyu_try1/items/0228870c41d9ac54e6e9
みたいなことの一環で、tensorflow 1.*に必要なものを削除したのかもしれません

cuda 10.0を入れたら動くだろうと、
https://stackoverflow.com/questions/58936927/tensorflow-1-14-on-google-collab-no-gpu
の質問の、
「Installing Cuda 10 (taken from the Tensorflow Docs)」
と書かれてるところよりも下に(二つのブロックに分かれて)記載されてるコードを全部実行したら、この質問のコードが実行できました
(途中で、キーボードのレイアウトを聞かれて、答えないといけないのがメンドくさいけど)

コードの下に「And updating the LD_LIBRARY_PATH 」と書かれてますが、それはやらなくて大丈夫でした

また、「Installing Cuda 10 (taken from the Tensorflow Docs)」よりも上に書かれてることも、やらなくて大丈夫でした

[riku_university]

!apt-get --purge remove cuda nvidia* libnvidia-*
!dpkg -l | grep cuda- | awk '{print $2}' | xargs -n1 dpkg --purge
!apt-get remove cuda-*
!apt autoremove

!wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
!sudo dpkg -i cuda-repo-ubuntu1804_10.0.130-1_amd64.deb
!sudo apt-key adv --fetch-keys https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
!sudo apt-get update
!wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64/nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
!sudo apt install -y ./nvidia-machine-learning-repo-ubuntu1804_1.0.0-1_amd64.deb
!sudo apt-get update

# Install NVIDIA driver
#!sudo apt-get install --no-install-recommends nvidia-driver-418
!sudo apt-get -y installnvidia-driver-418
# Reboot. Check that GPUs are visible using the command: nvidia-smi

# Install development and runtime libraries (~4GB)
#!sudo apt-get install --no-install-recommends \
!sudo apt-get install -y \
cuda-10-0 \
libcudnn7=7.6.2.24-1+cuda10.0 \
libcudnn7-dev=7.6.2.24-1+cuda10.0

# Install TensorRT. Requires that libcudnn7 is installed above.
# !sudo apt-get install -y --no-install-recommends libnvinfer5=5.1.5-1+cuda10.0 \
!sudo apt-get install -y libnvinfer5=5.1.5-1+cuda10.0 \
libnvinfer-dev=5.1.5-1+cuda10.0

!apt --fix-broken install

↑を実行したのですが、変わらずGPUを認識してくれません。

!wget https://battle.shawwn.com/sdc/stylegan2-imagenet-512/model.ckpt-533504.pkl
# 画像生成の実行
!python /content/stylegan2/run_generator.py generate-images \
--network=model.ckpt-533504.pkl \
--seeds=6600-6625 --truncation-psi=0.5

↑を実行するとエラーが変わり↓が表示されます。

tensorflow.python.framework.errors_impl.NotFoundError: /content/stylegan2/dnnlib/tflib/_cudacache/fused_bias_act_455e3ae619ac31ebc8962a246b0550da.so: undefined symbol: _ZN10tensorflow12OpDefBuilder5InputESs

[riku_university]

tensorflow のバージョンを1.14から1.15に変更することで解決しました。

[jbpb0]

> tensorflow のバージョンを1.14から1.15に変更することで解決しました。

あれ？
質問に記載のコードは

> !pip install tensorflow-gpu==1.15

ですが、質問のコードは、質問者さんが実際に実行してるコードと違うのでしょうか？

[riku_university]

解決策を検索して色々試していたのでその時に1.14に変更していたようです。
お騒がせしました。

Answer 1

2022-10-12 01:50:08.183797: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libcudart.so.10.0'; dlerror: libcudart.so.10.0: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/lib64-nvidia

cuda 10.0が無くなってますね

　
cuda 10.0を入れたら動くだろうと、
Tensorflow 1.14 on Google Collab - No GPU
の質問の、
「Installing Cuda 10 (taken from the Tensorflow Docs)」
と書かれてるところよりも下に(二つのブロックに分かれて)記載されてるコードを全部実行したら、この質問のコードが実行できました
(途中で、キーボードのレイアウトを聞かれて、答えないといけないのがメンドくさいけど)

コードの下に「And updating the LD_LIBRARY_PATH 」と書かれてますが、それはやらなくて大丈夫でした

また、「Installing Cuda 10 (taken from the Tensorflow Docs)」よりも上に書かれてることも、やらなくて大丈夫でした

　
【追記】
その後google colabのpythonが3.8に変わったので、現状ではpython 3.7に切り替えるために下記も実行する必要があります

!sudo apt-get install python3.7
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1
!sudo update-alternatives --config python3
!sudo apt install python3-pip

!python -m pip install --upgrade --force-reinstall pip

!pip install pillow
!pip install requests
!pip install protobuf==3.20

途中で下記のように聞かれるので、「2」を入力して「/usr/bin/python3.7」を選びます

  Selection    Path                Priority   Status
------------------------------------------------------------
* 0            /usr/bin/python3.8   2         auto mode
  1            /usr/bin/python3.6   1         manual mode
  2            /usr/bin/python3.7   1         manual mode
  3            /usr/bin/python3.8   2         manual mode

参考
How to install Python 3.7 in google colab?