dockerのmysqlで InnoDB: Write to file ./ibtmp1failed at offset XXX, XXX bytes should have been written, only XXX were writtenエラー

dockerで立てたmysqlに1G超えのダンプファイルをインポートしているとdockerが死んでいた。

再起動をすると、次のエラーが…

※ 複数台立ててたのを編集したので少し正確でないかもしれない。

 

$ docker-compose up
uStarting docker_my_app_mysql
Attaching to docker_my_app_mysql
my_app_mysql  | 2017-01-26T06:43:44.885861-00:00 0 [Note] mysqld (mysqld 5.7.10-log) starting as process 1 ...
my_app_mysql  | 2017-01-26T06:43:45.012334-00:00 0 [Warning] InnoDB: Using innodb_support_xa is deprecated and the parameter may be removed in future releases. Only innodb_support_xa=ON is allowed.
my_app_mysql  | 2017-01-26T06:43:45.012420-00:00 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
my_app_mysql  | 2017-01-26T06:43:45.012685-00:00 0 [Warning] InnoDB: Using innodb_file_format_max is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
my_app_mysql  | 2017-01-26T06:43:45.012816-00:00 0 [Note] InnoDB: PUNCH HOLE support available
my_app_mysql  | 2017-01-26T06:43:45.013025-00:00 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
my_app_mysql  | 2017-01-26T06:43:45.013061-00:00 0 [Note] InnoDB: Uses event mutexes
my_app_mysql  | 2017-01-26T06:43:45.013432-00:00 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
my_app_mysql  | 2017-01-26T06:43:45.013471-00:00 0 [Note] InnoDB: Compressed tables use zlib 1.2.8
my_app_mysql  | 2017-01-26T06:43:45.013689-00:00 0 [Note] InnoDB: Using Linux native AIO
my_app_mysql  | 2017-01-26T06:43:45.013924-00:00 0 [Note] InnoDB: Number of pools: 1
my_app_mysql  | 2017-01-26T06:43:45.074818-00:00 0 [Note] InnoDB: Using CPU crc32 instructions
my_app_mysql  | 2017-01-26T06:43:45.243777-00:00 0 [Note] InnoDB: Initializing buffer pool, total size = 256M, instances = 1, chunk size = 128M
my_app_mysql  | 2017-01-26T06:43:45.294167-00:00 0 [Note] InnoDB: Completed initialization of buffer pool
my_app_mysql  | 2017-01-26T06:43:45.334555-00:00 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
my_app_mysql  | 2017-01-26T06:43:45.460391-00:00 0 [Note] InnoDB: Highest supported file format is Barracuda.
my_app_mysql  | 2017-01-26T06:43:45.522417-00:00 0 [Note] InnoDB: Log scan progressed past the checkpoint lsn 120659067
my_app_mysql  | 2017-01-26T06:43:45.522560-00:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 120659076
my_app_mysql  | 2017-01-26T06:43:45.523474-00:00 0 [Note] InnoDB: Doing recovery: scanned up to log sequence number 120659076
my_app_mysql  | 2017-01-26T06:43:45.523570-00:00 0 [Note] InnoDB: Database was not shutdown normally!
my_app_mysql  | 2017-01-26T06:43:45.523939-00:00 0 [Note] InnoDB: Starting crash recovery.
my_app_mysql  | 2017-01-26T06:43:56.727224-00:00 0 [Note] InnoDB: Last MySQL binlog file position 0 1378506, file name mysql-bin.000053
my_app_mysql  | 2017-01-26T06:44:14.852856-00:00 0 [Note] InnoDB: Creating shared tablespace for temporary tables
my_app_mysql  | 2017-01-26T06:44:14.853217-00:00 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
my_app_mysql  | 2017-01-26T06:44:15.337430-00:00 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
my_app_mysql  | 2017-01-26T06:44:15.338084-00:00 0 [Note] InnoDB: 96 redo rollback segment(s) found. 96 redo rollback segment(s) are active.
my_app_mysql  | 2017-01-26T06:44:15.338349-00:00 0 [Note] InnoDB: 32 non-redo rollback segment(s) are active.
my_app_mysql  | 2017-01-26T06:44:15.340243-00:00 0 [Note] InnoDB: Waiting for purge to start
my_app_mysql  | 2017-01-26T06:44:15.344224-00:00 0 [Warning] InnoDB: 1048576 bytes should have been written. Only 954368 bytes written. Retrying for the remaining bytes.
my_app_mysql  | 2017-01-26T06:44:15.344389-00:00 0 [Warning] InnoDB: Retry attempts for writing partial data failed.
my_app_mysql  | 2017-01-26T06:44:15.344724-00:00 0 [ERROR] InnoDB: Write to file ./ibtmp1failed at offset 11534336, 1048576 bytes should have been written, only 954368 were written. Operating system error number 28. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded.
my_app_mysql  | 2017-01-26T06:44:15.344902-00:00 0 [ERROR] InnoDB: Error number 28 means 'No space left on device'
my_app_mysql  | 2017-01-26T06:44:15.345143-00:00 0 [Note] InnoDB: Some operating system error numbers are described at http://dev.mysql.com/doc/refman/5.7/en/operating-system-error-codes.html
my_app_mysql  | 2017-01-26T06:44:15.345268-00:00 0 [ERROR] InnoDB: Could not set the file size of './ibtmp1'. Probably out of disk space
my_app_mysql  | 2017-01-26T06:44:15.345501-00:00 0 [ERROR] InnoDB: Unable to create the shared innodb_temporary
my_app_mysql  | 2017-01-26T06:44:15.345535-00:00 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
my_app_mysql  | 2017-01-26T06:44:15.391007-00:00 0 [Note] InnoDB: page_cleaner: 1000ms intended loop took 30056ms. The settings might not be optimal. (flushed=0 and evicted=0, during the time.)
my_app_mysql  | 2017-01-26T06:44:15.392855-00:00 0 [Note] InnoDB: 5.7.10 started; log sequence number 120659076
my_app_mysql  | 2017-01-26T06:44:15.396314-00:00 0 [Note] Plugin 'FEDERATED' is disabled.
my_app_mysql  | 2017-01-26T06:44:15.396485-00:00 0 [Note] InnoDB: not started
my_app_mysql  | 2017-01-26T06:44:15.396545-00:00 0 [Note] InnoDB: Loading buffer pool(s) from /var/lib/mysql/ib_buffer_pool
my_app_mysql  | 2017-01-26T06:44:15.548692-00:00 0 [Note] Recovering after a crash using mysql-bin
my_app_mysql  | 2017-01-26T06:44:15.548898-00:00 0 [Note] Starting crash recovery...
my_app_mysql  | 2017-01-26T06:44:15.549285-00:00 0 [Note] Crash recovery finished.
my_app_mysql  | 06:44:15 UTC - mysqld got signal 11 ;
my_app_mysql  | This could be because you hit a bug. It is also possible that this binary
my_app_mysql  | or one of the libraries it was linked against is corrupt, improperly built,
my_app_mysql  | or misconfigured. This error can also be caused by malfunctioning hardware.
my_app_mysql  | Attempting to collect some information that could help diagnose the problem.
my_app_mysql  | As this is a crash and something is definitely wrong, the information
my_app_mysql  | collection process might fail.
my_app_mysql  |
my_app_mysql  | key_buffer_size=134217728
my_app_mysql  | read_buffer_size=2097152
my_app_mysql  | max_used_connections=0
my_app_mysql  | max_threads=300
my_app_mysql  | thread_count=0
my_app_mysql  | connection_count=0
my_app_mysql  | It is possible that mysqld could use up to
my_app_mysql  | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 3207063 K  bytes of memory
my_app_mysql  | Hope that's ok; if not, decrease some variables in the equation.
my_app_mysql  |
my_app_mysql  | Thread pointer: 0x0
my_app_mysql  | Attempting backtrace. You can use the following information to find out
my_app_mysql  | where mysqld died. If you see no messages after this, something went
my_app_mysql  | terribly wrong...
my_app_mysql  | stack_bottom = 0 thread_stack 0x30000
my_app_mysql  | mysqld(my_print_stacktrace+0x2c)[0xe4258c]
my_app_mysql  | mysqld(handle_fatal_signal+0x459)[0x782219]
my_app_mysql  | /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7f41168448d0]
my_app_mysql  | mysqld(thd_killed+0x23)[0xbe8723]
my_app_mysql  | mysqld(my_write+0x9a)[0xe4171a]
my_app_mysql  | mysqld(my_b_flush_io_cache+0x3d8)[0xe24fa8]
my_app_mysql  | mysqld(_ZN13MYSQL_BIN_LOG21sync_purge_index_fileEv+0x1d)[0xdd2b6d]
my_app_mysql  | mysqld(_ZN13MYSQL_BIN_LOG11open_binlogEPKcS1_mbbbP28Format_description_log_event+0xb2)[0xdd9c52]
my_app_mysql  | mysqld[0x77b2e9]
my_app_mysql  | mysqld(_Z11mysqld_mainiPPc+0x7b2)[0x77c132]
my_app_mysql  | /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7f411501eb45]
my_app_mysql  | mysqld[0x772e34]
my_app_mysql  | The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
my_app_mysql  | information that should help you find out what is causing the crash.
docker_my_app_mysql exited with code 2

アオーン。mysqlが起動してくれないよー

なんとなく、プロセスを確認してみる。

$ docker ps -a

CONTAINER ID  IMAGE         COMMAND                 CREATED        STATUS                     NAMES
3a7389377b6b  mysql:5.7.10  "/entrypoint.sh mysql"  2 minutes ago  Exited (1) 2 minutes ago   docker_my_app_mysql

みそは次。

[ERROR] InnoDB: Write to file ./ibtmp1failed at offset 11534336, 1048576 bytes should have been written, only 954368 were written. Operating system error number 28. Check that your OS and file system support files of this size. Check also that the disk is not full or a disk quota exceeded

ボリュームが肥大して悪さしているらしいので、削除する。

$ docker volume rm $(docker volume ls -qf dangling=true)
0124291639dc9d131c6778fa4bb81f564185c49efe9f50188fe21f1b3a66a225
$ docker-compose up
Starting docker_my_app_mysql
Attaching to docker_my_app_mysql
my_app_mysql  | Initializing database
my_app_mysql  | 2017-01-26T06:53:21.842049-00:00 0 [ERROR] --initialize specified but the data directory has files in it. Aborting.
my_app_mysql  | 2017-01-26T06:53:21.847704-00:00 0 [ERROR] Aborting
my_app_mysql  |
docker_my_app_mysql exited with code 1

まだ、起動しない。

プロセスも削除する。

$ docker rm 3a7389377b6b
3a7389377b6b

確認。

$ docker ps -a

CONTAINER ID  IMAGE         COMMAND                 CREATED        STATUS                     NAMES
$ docker-compose up
Creating docker_my_app_mysql
Attaching to docker_my_app_mysql
my_app_mysql  | Initializing database
my_app_mysql  | 2017-01-26T06:59:03.028503-00:00 0 [Warning] InnoDB: Using innodb_support_xa is deprecated and the parameter may be removed in future releases. Only innodb_support_xa=ON is allowed.
my_app_mysql  | 2017-01-26T06:59:03.035218-00:00 0 [Warning] InnoDB: Using innodb_file_format is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
my_app_mysql  | 2017-01-26T06:59:03.035369-00:00 0 [Warning] InnoDB: Using innodb_file_format_max is deprecated and the parameter may be removed in future releases. See http://dev.mysql.com/doc/refman/5.7/en/innodb-file-format.html
my_app_mysql  |  100 200 300
my_app_mysql  |  100 200
my_app_mysql  |  100 200
...(略)...
my_app_mysql  |
my_app_mysql  |
my_app_mysql  | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
my_app_mysql  |
my_app_mysql  | /entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
...(略)...
my_app_mysql  | 2017-01-26T07:01:08.599985-00:00 0 [Note] mysqld: Shutdown complete
my_app_mysql  |
my_app_mysql  |
my_app_mysql  | MySQL init process done. Ready for start up.
my_app_mysql  |
...(略)...
my_app_mysql  | Version: '5.7.10-log'  socket: '/var/lib/mysql/mysql.sock'  port: 3307  MySQL Community Server (GPL)
...(略)...

復活したぜ!

スポンサーリンク

シェアする

  • このエントリーをはてなブックマークに追加

フォローする

スポンサーリンク