Q&A
リジューム機能の検証を実施しています。
初回実行では、正常系でOracleDBのデータをBigqueryへバルクインサートしました。
２回目の実行では、異常系でバルクインサート中にネットワークを遮断させて敢えて失敗させます。
３回目にリジューム機能を使ったバルクインサートを実行すると失敗します。
embulk側の定義ファイルにオプション指定が不足しているのか？ 原因調査中となります。
環境：Windows7、OracleDB 12c、embulk 0.9.8
※2018/12/05更新
※Oracleテーブル定義 ----------------- -------- --- ID NOT NULL NUMBER(8) NUM NUMBER(12,2 STR CHAR(8) VARSTR VARCHAR2(8) DT DATE TIME0 TIMESTAMP(0 TIME6 TIMESTAMP(6 TIME9 TIMESTAMP(9 ※config_bq.yml.liquid in: type: oracle path_prefix: null driver_path: C:\Users\yazaki\Desktop\embulkDB\product\12.2.0\dbhome_1\sqldeveloper\jdbc\lib\ojdbc7.jar host: A0430-02396 user: system password: yazaki database: ORCL table: INPUT_TEST incremental: true incremental_columns: [id] skip_header_lines: 0 allow_extra_columns: false allow_optional_columns: true columns: - {name: ID, type: string} - {name: NUM, type: string} - {name: STR, type: string} - {name: VARSTR, type: string} - {name: DT, type: string} - {name: TIME0, type: timestamp, format: '%Y/%m/%d'} - {name: TIME6, type: timestamp, format: '%Y/%m/%d'} - {name: TIME9, type: timestamp, format: '%Y/%m/%d'} out: {type: bigquery, auth_method: json_key, json_keyfile: 'C:\Users\yazaki\Desktop\embulk\My First Project-6041ffecfba7.json', project: fluid-emissary-216806, dataset: embulk_test, auto_create_table: true, table: users, read_timeout_sec: 360000, open_timeout_sec: 360000, send_timeout_sec: 360000, mode: append} ※実行コマンド embulk-0.9.8.bat run config_bq.yml.liquid -c diff.yml >> C:\Users\yazaki\Desktop\embulk\log\ORA_to_BQ_log003.txt ※実行後のエラー org.embulk.exec.PartialExecutionException: org.embulk.spi.DataException: Column type 'double' set at incremental_columns option is not supported at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:339) at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:565) at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:34) at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:352) at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:349) at org.embulk.spi.Exec.doWith(Exec.java:22) at org.embulk.exec.BulkLoader.run(BulkLoader.java:349) at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:161) at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:292) at org.embulk.EmbulkRunner.run(EmbulkRunner.java:156) at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:436) at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:91) at org.embulk.cli.Main.main(Main.java:26) Caused by: org.embulk.spi.DataException: Column type 'double' set at incremental_columns option is not supported at org.embulk.input.jdbc.getter.AbstractColumnGetter.encodeToJson(org/embulk/input/jdbc/getter/AbstractColumnGetter.java:92) at org.embulk.input.jdbc.AbstractJdbcInputPlugin$LastRecordStore.accept(org/embulk/input/jdbc/AbstractJdbcInputPlugin.java:441) at org.embulk.input.jdbc.AbstractJdbcInputPlugin.run(org/embulk/input/jdbc/AbstractJdbcInputPlugin.java:490) at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor.runInputTask(org/embulk/exec/LocalExecutorPlugin.java:271) at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor.access$000(org/embulk/exec/LocalExecutorPlugin.java:196) at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor$1.call(org/embulk/exec/LocalExecutorPlugin.java:235) at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor$1.call(org/embulk/exec/LocalExecutorPlugin.java:232) at java.util.concurrent.FutureTask.run(java/util/concurrent/FutureTask) at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor$Worker) at java.lang.Thread.run(java/lang/Thread) Error: org.embulk.spi.DataException: Column type 'double' set at incremental_columns option is not supported
更新
※config_bq_incremental.yml in: type: oracle path_prefix: null driver_path: C:\Users\yazaki\Desktop\embulkDB\product\12.2.0\dbhome_1\sqldeveloper\jdbc\lib\ojdbc7.jar host: A0430-02396 user: system password: yazaki database: ORCL table: INPUT_TEST incremental: true incremental_columns: [id] column_options: ID: {type: string} NUM: {type: string} out: {type: bigquery, auth_method: json_key, json_keyfile: 'C:\Users\yazaki\Desktop\embulk\My First Project-6041ffecfba7.json', project: fluid-emissary-216806, dataset: embulk_test, auto_create_table: true, table: users, read_timeout_sec: 360000, open_timeout_sec: 360000, send_timeout_sec: 360000, mode: append} ※プレビュー C:\Users\yazaki\Desktop\embulk>embulk-0.9.8.bat preview config_bq_incremental.yml 2018-12-05 11:27:22.618 +0900: Embulk v0.9.8 2018-12-05 11:27:23.304 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected. 2018-12-05 11:27:25.457 +0900 [INFO] (main): Gem's home and path are set by default: "C:\Users\yazaki.embulk\lib\gems" 2018-12-05 11:27:26.190 +0900 [INFO] (main): Started Embulk v0.9.8 2018-12-05 11:27:26.268 +0900 [INFO] (0001:preview): Loaded plugin embulk-input-oracle (0.9.3) 2018-12-05 11:27:26.299 +0900 [INFO] (0001:preview): Connecting to jdbc:oracle:thin:@A0430-02396:1521:ORCL options {oracle.jdbc.ReadTimeout=1800000, user=system, password=***, oracle.net.CONNECT_TIMEOUT=300000} 2018-12-05 11:27:26.674 +0900 [INFO] (0001:preview): Using JDBC Driver 12.1.0.2.0 2018-12-05 11:27:26.783 +0900 [INFO] (0001:preview): Connecting to jdbc:oracle:thin:@A0430-02396:1521:ORCL options {oracle.jdbc.ReadTimeout=1800000, user=system, password=***, oracle.net.CONNECT_TIMEOUT=300000} 2018-12-05 11:27:26.845 +0900 [INFO] (0001:preview): SQL: SELECT * FROM "INPUT_TEST" ORDER BY "ID" 2018-12-05 11:27:26.861 +0900 [INFO] (0001:preview): > 0.02 seconds +-----------+------------+------------+---------------+-------------------------+-------------------------+-----------------------------+-----------------------------+ | ID:string | NUM:string | STR:string | VARSTR:string | DT:timestamp | TIME0:timestamp | TIME6:timestamp | TIME9:timestamp | +-----------+------------+------------+---------------+-------------------------+-------------------------+-----------------------------+-----------------------------+ | 179510 | 81596323 | OYYSHALN | PMTVRDKR | 2018-12-04 08:57:26 UTC | 2018-12-04 08:57:27 UTC | 2018-12-04 08:57:26.922 UTC | 2018-12-04 08:57:26.922 UTC | | 179511 | 45967077 | EDMHMKSA | HCQMWTDO | 2018-12-04 08:57:26 UTC | 2018-12-04 08:57:27 UTC | 2018-12-04 08:57:26.969 UTC | 2018-12-04 08:57:26.969 UTC | | 179512 | 45306398 | ZTGYAJZG | JGMEIMTI | 2018-12-04 08:57:26 UTC | 2018-12-04 08:57:27 UTC | 2018-12-04 08:57:26.969 UTC | 2018-12-04 08:57:26.969 UTC | ※実行コマンド C:\Users\yazaki\Desktop\embulk>embulk-0.9.8.bat run config_bq_incremental.yml -c diff.yml >> C:\Users\yazaki\Desktop\embulk\log\ORA_to_BQ_log004.txt ※実行後のエラー org.embulk.exec.PartialExecutionException: org.embulk.spi.DataException: Column type 'string' set at incremental_columns option is not supported at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:339) at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:565) at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:34) at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:352) at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:349) at org.embulk.spi.Exec.doWith(Exec.java:22) at org.embulk.exec.BulkLoader.run(BulkLoader.java:349) at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:161) at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:292) at org.embulk.EmbulkRunner.run(EmbulkRunner.java:156) at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:436) at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:91) at org.embulk.cli.Main.main(Main.java:26) Caused by: org.embulk.spi.DataException: Column type 'string' set at incremental_columns option is not supported at org.embulk.input.jdbc.getter.AbstractColumnGetter.encodeToJson(org/embulk/input/jdbc/getter/AbstractColumnGetter.java:92) at org.embulk.input.jdbc.AbstractJdbcInputPlugin$LastRecordStore.accept(org/embulk/input/jdbc/AbstractJdbcInputPlugin.java:441) at org.embulk.input.jdbc.AbstractJdbcInputPlugin.run(org/embulk/input/jdbc/AbstractJdbcInputPlugin.java:490) at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor.runInputTask(org/embulk/exec/LocalExecutorPlugin.java:271) at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor.access$000(org/embulk/exec/LocalExecutorPlugin.java:196) at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor$1.call(org/embulk/exec/LocalExecutorPlugin.java:235) at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor$1.call(org/embulk/exec/LocalExecutorPlugin.java:232) at java.util.concurrent.FutureTask.run(java/util/concurrent/FutureTask) at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor) at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor$Worker) at java.lang.Thread.run(java/lang/Thread) Error: org.embulk.spi.DataException: Column type 'string' set at incremental_columns option is not supported
回答1件
0
ベストアンサー
リジュームは動かないので使わないでください。
投稿2018/11/28 11:54
総合スコア415
あなたの回答
tips
プレビュー
バッドをするには、ログインかつ
こちらの条件を満たす必要があります。
2018/11/28 23:49
2018/11/29 00:03
2018/11/29 03:21
2018/11/29 05:46
2018/11/29 09:31
2018/11/30 00:38
2018/11/30 08:47
2018/12/04 02:56
2018/12/04 03:12
2018/12/04 08:00
2018/12/04 09:06
2018/12/04 09:28
2018/12/04 12:15
2018/12/05 01:06
2018/12/05 01:10
2018/12/05 01:44
2018/12/05 02:16
2018/12/05 02:30
2018/12/05 05:29
2018/12/05 05:35