Wordpressのデータベース接続確率エラー

Question

問題点

AWS上にwordpressの環境を構築したのですが、データベース接続エラーにより管理画面にアクセスできなくなりました。

エラー内容

mariadb.log

2021-03-19 17:00:56 140449615568704 [Note] RocksDB: 2 column families found
2021-03-19 17:00:56 140449615568704 [Note] RocksDB: Column Families at start:
2021-03-19 17:00:56 140449615568704 [Note]   cf=default
2021-03-19 17:00:56 140449615568704 [Note]     write_buffer_size=67108864
2021-03-19 17:00:56 140449615568704 [Note]     target_file_size_base=67108864
2021-03-19 17:00:56 140449615568704 [Note]   cf=__system__
2021-03-19 17:00:56 140449615568704 [Note]     write_buffer_size=67108864
2021-03-19 17:00:56 140449615568704 [Note]     target_file_size_base=67108864
2021-03-19 17:00:56 140449615568704 [Note] RocksDB: Table_store: loaded DDL data for 0 tables
2021-03-19 17:00:56 140449615568704 [Note] RocksDB: global statistics using get_sched_indexer_t indexer
2021-03-19 17:00:56 140449615568704 [Note] MyRocks storage engine plugin has been successfully initialized.
Fri Mar 19 17:00:56 2021 PerconaFT recovery starting in env /var/lib/mysql/
Fri Mar 19 17:00:56 2021 PerconaFT recovery scanning backward from 121384
Fri Mar 19 17:00:56 2021 PerconaFT recovery bw_end_checkpoint at 121384 timestamp 1616140842975886 xid 121380 (bw_newer)
Fri Mar 19 17:00:56 2021 PerconaFT recovery bw_begin_checkpoint at 121380 timestamp 1616140842975228 (bw_between)
Fri Mar 19 17:00:56 2021 PerconaFT recovery turning around at begin checkpoint 121380 time 658
Fri Mar 19 17:00:56 2021 PerconaFT recovery starts scanning forward to 121384 from 121380 left 4 (fw_between)
Fri Mar 19 17:00:56 2021 PerconaFT recovery closing 2 dictionaries
Fri Mar 19 17:00:56 2021 PerconaFT recovery making a checkpoint
Fri Mar 19 17:00:56 2021 PerconaFT recovery done
2021-03-19 17:00:56 140449615568704 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-03-19 17:00:56 140449615568704 [Note] InnoDB: Uses event mutexes
2021-03-19 17:00:56 140449615568704 [Note] InnoDB: Compressed tables use zlib 1.2.7
2021-03-19 17:00:56 140449615568704 [Note] InnoDB: Using Linux native AIO
2021-03-19 17:00:56 140449615568704 [Note] InnoDB: Number of pools: 1
2021-03-19 17:00:56 140449615568704 [Note] InnoDB: Using SSE2 crc32 instructions
2021-03-19 17:00:56 140449615568704 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-03-19 17:00:56 140449615568704 [ERROR] InnoDB: mmap(137560064 bytes) failed; errno 12
2021-03-19 17:00:56 140449615568704 [ERROR] InnoDB: Cannot allocate memory for the buffer pool
2021-03-19 17:00:56 140449615568704 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2021-03-19 17:00:56 140449615568704 [Note] InnoDB: Starting shutdown...

試したこと

[ERROR] InnoDB: Cannot allocate memory for the buffer pool
メモリが足りてないといわれていたので以下参考にswap領域を作成し有効化しました
https://tech.librastudio.co.jp/entry/index.php/2018/05/22/post-1891/

   #free
           total        used        free      shared  buff/cache   available
Mem:        1006900      424820      400324         360      181756      443208
Swap:       1006896        1792     1005104
コード

いざmariaDBを起動してみましたが、

#sudo service mariadb start

Redirecting to /bin/systemctl start mariadb.service
Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.

起動できない・・・

もう一度ログを確認

2021-03-19 17:41:56 139824757690176 [Note] RocksDB: 2 column families found
2021-03-19 17:41:56 139824757690176 [Note] RocksDB: Column Families at start:
2021-03-19 17:41:56 139824757690176 [Note]   cf=default
2021-03-19 17:41:56 139824757690176 [Note]     write_buffer_size=67108864
2021-03-19 17:41:56 139824757690176 [Note]     target_file_size_base=67108864
2021-03-19 17:41:56 139824757690176 [Note]   cf=__system__
2021-03-19 17:41:56 139824757690176 [Note]     write_buffer_size=67108864
2021-03-19 17:41:56 139824757690176 [Note]     target_file_size_base=67108864
2021-03-19 17:41:56 139824757690176 [ERROR] RocksDB: Error opening instance, Status Code: 5, Status: IO error: While lock file: ./.rocksdb/LOCK: Resource temporarily unavailable
2021-03-19 17:41:56 139824757690176 [ERROR] Plugin 'ROCKSDB' init function returned error.
2021-03-19 17:41:56 139824757690176 [ERROR] Plugin 'ROCKSDB' registration as a STORAGE ENGINE failed.
Couldn't start tokuft because some other tokuft process is using the same directory [/var/lib/mysql/] for [environment]
2021-03-19 17:41:56 139824757690176 [ERROR] TokuDB unknown error 11
2021-03-19 17:41:56 139824757690176 [ERROR] Plugin 'TokuDB' init function returned error.
2021-03-19 17:41:56 139824757690176 [ERROR] Plugin 'TokuDB' registration as a STORAGE ENGINE failed.
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Uses event mutexes
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Compressed tables use zlib 1.2.7
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Using Linux native AIO
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Number of pools: 1
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Using SSE2 crc32 instructions
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Completed initialization of buffer pool
2021-03-19 17:41:56 139823908087552 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Highest supported file format is Barracuda.
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: 128 out of 128 rollback segments are active.
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Creating shared tablespace for temporary tables
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: Waiting for purge to start
2021-03-19 17:41:56 139824757690176 [Note] InnoDB: 5.7.20 started; log sequence number 115031297
2021-03-19 17:41:56 139823897597696 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
2021-03-19 17:41:56 139823897597696 [Note] InnoDB: Buffer pool(s) load completed at 210319 17:41:56
2021-03-19 17:41:56 139824757690176 [Note] Plugin 'FEEDBACK' is disabled.
2021-03-19 17:41:56 139824757690176 [Warning] mysqld: GSSAPI plugin : default principal 'mariadb/ip-172-31-28-115.ap-northeast-1.compute.internal@' not found in keytab
2021-03-19 17:41:56 139824757690176 [ERROR] mysqld: Server GSSAPI error (major 851968, minor 2529639093) : gss_acquire_cred failed -Unspecified GSS failure.  Minor code may provide more information. Keytab FILE:/etc/krb5.keytab is nonexistent or empty. 
2021-03-19 17:41:56 139824757690176 [ERROR] Plugin 'gssapi' init function returned error.
2021-03-19 17:41:56 139824757690176 [Note] Recovering after a crash using tc.log
2021-03-19 17:41:56 139824757690176 [ERROR] Recovery failed! You must enable all engines that were enabled at the moment of the crash
2021-03-19 17:41:56 139824757690176 [ERROR] Crash recovery failed. Either correct the problem (if it's, for example, out of memory error) and restart, or delete tc log and start mysqld with --tc-heuristic-recover={commit|rollback}
2021-03-19 17:41:56 139824757690176 [ERROR] Can't init tc log
2021-03-19 17:41:56 139824757690176 [ERROR] Aborting

また別のエラー？
出てるエラーについてググっても対処法が見つかりませんでした。

詳しい方いましたらご教授願います。

Answer 1

自己解決いたしました

[ERROR] Can't init tc log

上記のようにtc.logが初期化されていないといったエラーから
/var/lib/mysql/tc.logを削除することでmysqlへの接続ができるようになりました。
はっきりとした原因はわかりませんがしばらく様子を見てみます。

ありがとうございました。

参考サイト
https://analyzegear.co.jp/blog/442

Comments

[退会済みユーザー]

暫定的な解決、おめでとうございます。
お役に立てず申し訳ないです。
後半部分のエラーの内容が違っていましたね。
もう少し時間を割くべきでした。
重ね重ね申し訳ないです。

スワップファイルが必要になるということはスペックが足りていないということなので、
もし可能であればスペックを上げた方が良いかもしれません。
無事に運用が続けられることをお祈り申し上げます。

Answer 2

英語のリンクで申し訳ないのですが、
「MariaDBの前のデータが残っており実行中だからエラーが出る」
「構成ファイルを取り除いたら再スタートできた」
というような文言が見て取れます。

logもほぼ同じように見えますので、同じケースのエラーではないかと愚考いたします。
翻訳ツールなどでご一読いただけましたら幸いでございます。

参考URL　MariaDB　Mariab not hanging when trying start
https://mariadb.com/kb/en/mariab-not-hanging-when-trying-start/

参考URL　Red Hat Bugzilla MariaDB fails to start after upgrade to F29
https://bugzilla.redhat.com/show_bug.cgi?id=1640910#c4

以下蛇足です。
日本語で資料を出してくれる人がいたらそちらを参考にして下さいね。

Comments

[castle]

回答ありがとうございます

上記参考に/etc/my.cnf.d/local.cnf（自分の環境では/etc/my.cnf.d/mariasb-server.cnf）に設定確認しましたが問題となっているinnodb_file_format=Barracudaの設定は行っていませんでした。

[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
log-error=/var/log/mariadb/mariadb.log
pid-file=/run/mariadb/mariadb.pid
innodb_buffer_pool_size=1000M