oracleDBのデータをBigQueryへ増分バルクインサートしたいのですが
embulkで増分対象とするデータを検索する際にある2つのテーブルを結合させて
条件にマッチングしたレコードのみを対象としたいです。
use_raw_query_with_incrementalオプションを使って、検証しているのですが
incremental_columnsオプションが有効にならずエラーを出力しております。
原因調査中となります。
環境:Win7、embulk 0.9.8 、oracle12c
SQL> desc INPUT_TEST_001 名前 NULL? 型 ------------------------ -------- ------------- BANGOU NOT NULL NUMBER(8) NUM NUMBER(12,2) STR CHAR(8) VARSTR VARCHAR2(8) DT DATE TIME0 TIMESTAMP(0) TIME6 TIMESTAMP(6) TIME9 TIMESTAMP(9) SQL> desc INPUT_TEST_002 名前 NULL? 型 ------------------------ -------- ------------- ID NOT NULL NUMBER(8) NUM NUMBER(12,2) STR CHAR(8) VARSTR VARCHAR2(8) DT DATE TIME0 TIMESTAMP(0) TIME6 TIMESTAMP(6) TIME9 TIMESTAMP(9) ※C:\Users\xxxxx\Desktop\embulk\config_bq_incremental.yml.liquid in: type: oracle path_prefix: null driver_path: C:\Users\xxxxx\Desktop\embulkDB\product\12.2.0\dbhome_1\sqldeveloper\jdbc\lib\ojdbc7.jar host: xxxx-xxxx user: system password: xxxxx database: ORCL query: select A1.BANGOU,A1.NUM,A1.STR from INPUT_TEST_001 A1,INPUT_TEST_002 B1 where A1.NUM = B1.NUM incremental: true incremental_columns: [dt] use_raw_query_with_incremental: true last_record: [1] out: {type: bigquery, auth_method: json_key, json_keyfile: 'C:\Users\xxxxx\Desktop\embulk\My First Project-6041ffecfba7.json', project: xxxxxxxx, dataset: embulk_test, auto_create_table: true, table: INPUT_TEST_001, read_timeout_sec: 360000, open_timeout_sec: 360000, send_timeout_sec: 360000, mode: append} C:\Users\xxxxx>C:\Users\xxxxx\Desktop\embulk\embulk-0.9.8.bat run C:\Users\xxxxx\Desktop\embulk\config_bq_incremental.yml.liquid -c C:\Users\xxxxx\Desktop\embulk\diff_001.yml 2018-12-10 14:47:35.047 +0900: Embulk v0.9.8 2018-12-10 14:47:36.014 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected. 2018-12-10 14:47:39.714 +0900 [INFO] (main): Gem's home and path are set by default: "C:\Users\xxxxx.embulk\lib\gems" 2018-12-10 14:47:43.275 +0900 [INFO] (main): Started Embulk v0.9.8 2018-12-10 14:47:43.384 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-oracle (0.9.3) 2018-12-10 14:47:48.941 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.4.9) 2018-12-10 14:47:48.973 +0900 [INFO] (0001:transaction): Connecting to jdbc:oracle:thin:@xxxx-xxxx:1521:ORCL options {oracle.jdbc.ReadTimeout=1800000, user=system, password=***, oracle.net.CONNECT_TIMEOUT=300000} 2018-12-10 14:47:49.503 +0900 [INFO] (0001:transaction): Using JDBC Driver 12.1.0.2.0 org.embulk.exec.PartialExecutionException: org.embulk.config.ConfigException: Column ":dt" doesn't exist in query string at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:339) at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:565) at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:34) at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:352) at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:349) at org.embulk.spi.Exec.doWith(Exec.java:22) at org.embulk.exec.BulkLoader.run(BulkLoader.java:349) at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:161) at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:292) at org.embulk.EmbulkRunner.run(EmbulkRunner.java:156) at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:436) at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:91) at org.embulk.cli.Main.main(Main.java:26) Suppressed: java.lang.NullPointerException at org.embulk.exec.BulkLoader.doCleanup(BulkLoader.java:462) at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:396) at org.embulk.exec.BulkLoader$3.run(BulkLoader.java:393) at org.embulk.spi.Exec.doWith(Exec.java:22) at org.embulk.exec.BulkLoader.cleanup(BulkLoader.java:393) at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:164) ... 5 more Caused by: org.embulk.config.ConfigException: Column ":dt" doesn't exist in query string at org.embulk.input.jdbc.AbstractJdbcInputPlugin.getRawQuery(AbstractJdbcInputPlugin.java:365) at org.embulk.input.jdbc.AbstractJdbcInputPlugin.setupTask(AbstractJdbcInputPlugin.java:217) at org.embulk.input.jdbc.AbstractJdbcInputPlugin.transaction(AbstractJdbcInputPlugin.java:201) at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:506) ... 11 more Error: org.embulk.config.ConfigException: Column ":dt" doesn't exist in query string C:\Users\xxxxx>
回答4件
あなたの回答
tips
プレビュー
下記のような回答は推奨されていません。
このような回答には修正を依頼しましょう。
2018/12/14 04:32