編集履歴

質問編集履歴

質問更新

2019/04/16 08:02

投稿

raccoondog

スコア77

title CHANGED Viewed

File without changes

body CHANGED Viewed

@@ -77,10 +77,126 @@
 Error: org.jruby.exceptions.RaiseException: (Error) failed during waiting a Copy job, get_job(yzk-gcp-project, embulk_copy_job_ec19947a-757f-4119-9620-7ccdeb023777), errors:[{:reason=>"invalid", :message=>"Provided Schema does not match Table yzk-gcp-project:srdcs_data.T_RS_TRAN_20190416. Field ORDER_NO has changed type from NUMERIC to STRING"}]
+●embulk定義
+in:
+  type: oracle
+  path_prefix: null
+  driver_path: /opt/oracle/product/12.1.0/db/jdbc/lib/ojdbc7.jar
+  host: srdcs04
+  user: srdcs
+  password: srdcs
+  database: srdcsc
+  column_options:
-oracle⇒embulk⇒embulk_plugin⇒bigquery【処理結果】
+     ORDER_NO: { value_type: string }
+  query: |
+     select
+        trs.ORDER_NO,
+        trs.RS_TYPE,
+        trs.FACTORY_CODE,
+        trs.ORG_CODE,
+        trs.TRAN_DATE,
+        trs.HINBAN,
+        trs.QTY,
+        trs.SLIP_NO,
+        trs.ITEM_NO,
+        trs.DIVISION_NO,
+        trs.FROM_FACTORY_CODE,
+        trs.FROM_ORG_CODE,
+        trs.FROM_SUBINV_CODE,
+        trs.FROM_VALUE_CODE,
+        trs.FROM_OPERATOR_NAME,
+        trs.TO_FACTORY_CODE,
+        trs.TO_ORG_CODE,
+        trs.TO_SUBINV_CODE,
+        trs.TO_VALUE_CODE,
+        trs.TO_OPERATOR_NAME,
+        trs.SUB_SYSTEM_TYPE1,
+        trs.CANCEL_FLAG,
+        trs.ACT_CREATION_DATE,
+        trs.WH_ACT_CREATION_DATE,
+        trs.JOB_CODE,
+        trs.UNIT_PRICE,
+        trs.ERROR_STATUS,
+        trs.DESCRIPTION,
+        trs.LOCATION,
+        trs.REVISION_NO,
+        trs.OPE_CODE,
+        trs.BOX_KIND,
+        trs.DEMAND_NO,
+        trs.SUB_NO,
+        trs.TRAFFIC_NO,
+        trs.IF_FLAG,
+        trs.SELL_BUY_DATA_FLAG,
+        trs.SOURCE_ORDER_NO,
+        trs.FT_DEFINE_ID,
+        trs.FROM_SELL_BUY_ID,
+        trs.TO_SELL_BUY_ID,
+        trs.CREATION_DATE,
+        trs.CREATED_BY,
+        trs.LAST_UPDATE_DATE,
+        trs.LAST_UPDATED_BY,
+        trs.UPDATE_REQUEST_ID,
+        trs.TIME_STAMP,
+        trs.INV_UPDATE_FLAG,
+        trs.ISSUE_SEQ,
+        trs.SOUMEI_ID,
+        trs.GCPCS_IF_FLAG,
+        trs.GCPCS_IF_CREATION_DATE,
+        trs.RECIEPT_CODE,
+        trs.DELIV_BIN_NO,
+        trs.DELIV_NO,
+        trs.RECV_BIN_NO,
+        trs.SUB_SYSTEM_TYPE2,
+        trs.SHIP_DATE,
+        trs.C_NO,
+        trs.T_NO,
+        trs.RECV_OFFICE_CODE,
+        trs.SR_IF_FLAG,
+        trs.SR_IF_CREATION_DATE,
+        trs.PS_IF_FLAG,
+        trs.PS_IF_CREATION_DATE,
+        trs.CUSTOMER_CODE,
+        trs.KS_type,
+        trs.SUPPLY_SECTION,
+        trs.MANAGE_NO,
+        trs.SEND_OFFICE_CODE,
+        trs.SOUMEI_SLIP_NO,
+        trs.SOUMEI_ITEM_NO,
+        trs.PS_YAZAKI_HINBAN,
+        trs.INVOICE_CASE_INFO_NO,
+        trs.GYK_SITE_CODE,
+        trs.CASE_NO,
+        trs.CASE_SUB_NO,
+        trs.FROM_LOCATION,
+        trs.TO_LOCATION,
+        trs.GYK_TRAN_type,
+        trs.INV_INFO_UPDATE_FLAG,
+        (select code_description from m_code where code_type = 'rs_type' and code  = trs.rs_type) rs_name,
+        (select org_name from m_org where org_code = trs.org_code ) org_name,
+        (select factory_name from m_factory where factory_code = trs.from_factory_code ) from_factory_name,
+        (select org_name from m_org where org_code = trs.from_org_code ) from_org_name,
+        (select subinv_name from m_subinv where org_code = trs.from_org_code and subinv_code = trs.from_subinv_code) from_subinv_name,
+        (select factory_name from m_factory where factory_code = trs.to_factory_code ) to_factory_name,
+        (select org_name from m_org where org_code = trs.to_org_code ) to_org_name,
+        (select subinv_name from m_subinv where org_code = trs.to_org_code and subinv_code = trs.to_subinv_code) to_subinv_name,
+        (select ope_code_name from m_ope_code where factory_code = trs.factory_code and ope_code = trs.ope_code) ope_code_name,
+        (select sub_system_name1 from m_sub_system_type1 where sub_system_type1 = trs.sub_system_type1) sub_system_name1,
+        (select code_description  from m_code where code_type= 'cancel_flag' and code= trs.cancel_flag) cancel_name,
+        (select sub_system_name2 from m_sub_system_type2 where sub_system_type1 = trs.sub_system_type1 and sub_system_type2 = trs.sub_system_type2) sub_system_name2
+     from
+        T_RS_TRAN trs
+     where
+        IF_FLAG = '1' and TO_DATE(TRAN_DATE) >= TO_DATE('20190226','yyyyMMdd') and TO_DATE(TRAN_DATE) <= TO_DATE('20190227','yyyyMMdd')
+out: {
+type: bigquery,
+auth_method: json_key,
+json_keyfile: '/home1/bigquery/conf/My First Project-6041ffecfba7.json',
-NUMBER⇒double※embulkにより自動変換⇒float⇒float【●】
+project: fluid-emissary-216806,
-NUMBER⇒double※embulkにより自動変換⇒float⇒NUMERIC【×】
+dataset: embulk_test,
+auto_create_table: true,
+table: T_RS_TRAN,
-NUMBER⇒long※embulkの定義で強制変換⇒integer⇒NUMERIC【×】
+open_read_timeout_sec: 360000,
-NUMBER⇒string※embulkの定義で強制変換⇒string⇒NUMERIC【×】
+send_timeout_sec: 360000,
+read_timeout_sec: 360000}
 ```

状況更新

2019/04/16 08:01

投稿

raccoondog

スコア77

title CHANGED Viewed

File without changes

body CHANGED Viewed

@@ -22,6 +22,61 @@
 ※2019/4/16
 ```ここに言語を入力
+環境：oracle 12c、embulk 0.9.15
+※Oracle定義
+SQL> desc T_RS_TRAN
+ 名前                                      NULL?    型
+ ----------------------------------------- -------- ----------------------------
+ ORDER_NO                                  NOT NULL NUMBER(15)
+ RS_TYPE                                   NOT NULL VARCHAR2(1)
+ FACTORY_CODE                              NOT NULL VARCHAR2(4)
+ ORG_CODE                                  NOT NULL VARCHAR2(3)
+ TRAN_DATE                                 NOT NULL DATE
+ HINBAN                                    NOT NULL VARCHAR2(40)
+ QTY                                       NOT NULL NUMBER(11,2)
+ SLIP_NO                                   NOT NULL VARCHAR2(10)
+ ITEM_NO                                   NOT NULL VARCHAR2(2)
+ DIVISION_NO                                        NUMBER(1)
+ ●●
+ ●●
+ ●●
+ INV_INFO_UPDATE_FLAG                               VARCHAR2(1)
+※実行ログ
+2019-04-16 16:47:24.274 +0900 [ERROR] (0001:transaction): embulk-output-bigquery: failed during waiting a Copy job, get_job(yzk-gcp-project, embulk_copy_job_ec19947a-757f-4119-9620-7ccdeb023777), errors:[{:reason=>"invalid", :message=>"Provided Schema does not match Table yzk-gcp-project:srdcs_data.T_RS_TRAN_20190416. Field ORDER_NO has changed type from NUMERIC to STRING"}]
+2019-04-16 16:47:24.276 +0900 [INFO] (0001:transaction): embulk-output-bigquery: Delete table... yzk-gcp-project:srdcs_data.LOAD_TEMP_eb1cab64_4892_425b_bf29_d279cc655440_T_RS_TRAN_20190416
+2019-04-16 16:47:24.590 +0900 [INFO] (0001:transaction): embulk-output-bigquery: delete /var/tmp/embulk_output_bigquery_20190416-15487-1vrnvze.15487.2004.csv
+2019-04-16 16:47:24.590 +0900 [INFO] (0001:transaction): embulk-output-bigquery: delete /var/tmp/embulk_output_bigquery_20190416-15487-1vrnvze.15487.2006.csv
+2019-04-16 16:47:24.591 +0900 [INFO] (0001:transaction): embulk-output-bigquery: delete /var/tmp/embulk_output_bigquery_20190416-15487-1vrnvze.15487.2008.csv
+2019-04-16 16:47:24.591 +0900 [INFO] (0001:transaction): embulk-output-bigquery: delete /var/tmp/embulk_output_bigquery_20190416-15487-1vrnvze.15487.2010.csv
+2019-04-16 16:47:24.591 +0900 [INFO] (0001:transaction): embulk-output-bigquery: delete /var/tmp/embulk_output_bigquery_20190416-15487-1vrnvze.15487.2012.csv
+org.embulk.exec.PartialExecutionException: org.jruby.exceptions.RaiseException: (Error) failed during waiting a Copy job, get_job(yzk-gcp-project, embulk_copy_job_ec19947a-757f-4119-9620-7ccdeb023777), errors:[{:reason=>"invalid", :message=>"Provided Schema does not match Table yzk-gcp-project:srdcs_data.T_RS_TRAN_20190416. Field ORDER_NO has changed type from NUMERIC to STRING"}]
+        at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:340)
+        at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:566)
+        at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:35)
+        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:353)
+        at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:350)
+        at org.embulk.spi.Exec.doWith(Exec.java:22)
+        at org.embulk.exec.BulkLoader.run(BulkLoader.java:350)
+        at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:178)
+        at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:292)
+        at org.embulk.EmbulkRunner.run(EmbulkRunner.java:156)
+        at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:436)
+        at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:91)
+        at org.embulk.cli.Main.main(Main.java:64)
+Caused by: org.jruby.exceptions.RaiseException: (Error) failed during waiting a Copy job, get_job(yzk-gcp-project, embulk_copy_job_ec19947a-757f-4119-9620-7ccdeb023777), errors:[{:reason=>"invalid", :message=>"Provided Schema does not match Table yzk-gcp-project:srdcs_data.T_RS_TRAN_20190416. Field ORDER_NO has changed type from NUMERIC to STRING"}]
+        at RUBY.wait_load(/export/home/srdcs/.embulk/lib/gems/gems/embulk-output-bigquery-0.4.9/lib/embulk/output/bigquery/bigquery_client.rb:346)
+        at RUBY.block in copy(/export/home/srdcs/.embulk/lib/gems/gems/embulk-output-bigquery-0.4.9/lib/embulk/output/bigquery/bigquery_client.rb:289)
+        at RUBY.with_job_retry(/export/home/srdcs/.embulk/lib/gems/gems/embulk-output-bigquery-0.4.9/lib/embulk/output/bigquery/bigquery_client.rb:58)
+        at RUBY.copy(/export/home/srdcs/.embulk/lib/gems/gems/embulk-output-bigquery-0.4.9/lib/embulk/output/bigquery/bigquery_client.rb:249)
+        at RUBY.transaction(/export/home/srdcs/.embulk/lib/gems/gems/embulk-output-bigquery-0.4.9/lib/embulk/output/bigquery.rb:407)
+        at RUBY.transaction(uri:classloader:/gems/embulk-0.9.15-java/lib/embulk/output_plugin.rb:64)
+Error: org.jruby.exceptions.RaiseException: (Error) failed during waiting a Copy job, get_job(yzk-gcp-project, embulk_copy_job_ec19947a-757f-4119-9620-7ccdeb023777), errors:[{:reason=>"invalid", :message=>"Provided Schema does not match Table yzk-gcp-project:srdcs_data.T_RS_TRAN_20190416. Field ORDER_NO has changed type from NUMERIC to STRING"}]
 oracle⇒embulk⇒embulk_plugin⇒bigquery【処理結果】
 NUMBER⇒double※embulkにより自動変換⇒float⇒float【●】

質問更新

2019/04/16 07:47

投稿

raccoondog

スコア77

title CHANGED Viewed

File without changes

body CHANGED Viewed

@@ -18,4 +18,14 @@
 ```
-※OracleのVIEWは使用せず、対応したいです。
+※OracleのVIEWは使用せず、対応したいです。
+※2019/4/16
+```ここに言語を入力
+oracle⇒embulk⇒embulk_plugin⇒bigquery【処理結果】
+NUMBER⇒double※embulkにより自動変換⇒float⇒float【●】
+NUMBER⇒double※embulkにより自動変換⇒float⇒NUMERIC【×】
+NUMBER⇒long※embulkの定義で強制変換⇒integer⇒NUMERIC【×】
+NUMBER⇒string※embulkの定義で強制変換⇒string⇒NUMERIC【×】
+```

質問更新

2019/04/16 07:41

投稿

raccoondog

スコア77

title CHANGED Viewed

File without changes

body CHANGED Viewed

@@ -16,4 +16,6 @@
 ↓自動解析
 bigquery:float
-```
+```
+※OracleのVIEWは使用せず、対応したいです。

質問内容更新

2019/04/10 00:40

投稿

raccoondog

スコア77

title CHANGED Viewed

File without changes

body CHANGED Viewed

File without changes