質問編集履歴
3
質問内容の更新
title
CHANGED
File without changes
|
body
CHANGED
@@ -89,4 +89,83 @@
|
|
89
89
|
|
90
90
|
Error: org.embulk.spi.DataException: Column type 'double' set at incremental_columns option is not supported
|
91
91
|
|
92
|
+
```
|
93
|
+
|
94
|
+
更新
|
95
|
+
```ここに言語を入力
|
96
|
+
※config_bq_incremental.yml
|
97
|
+
in:
|
98
|
+
type: oracle
|
99
|
+
path_prefix: null
|
100
|
+
driver_path: C:\Users\yazaki\Desktop\embulkDB\product\12.2.0\dbhome_1\sqldeveloper\jdbc\lib\ojdbc7.jar
|
101
|
+
host: A0430-02396
|
102
|
+
user: system
|
103
|
+
password: yazaki
|
104
|
+
database: ORCL
|
105
|
+
table: INPUT_TEST
|
106
|
+
incremental: true
|
107
|
+
incremental_columns: [id]
|
108
|
+
column_options:
|
109
|
+
ID: {type: string}
|
110
|
+
NUM: {type: string}
|
111
|
+
out: {type: bigquery, auth_method: json_key, json_keyfile: 'C:\Users\yazaki\Desktop\embulk\My
|
112
|
+
First Project-6041ffecfba7.json', project: fluid-emissary-216806, dataset: embulk_test,
|
113
|
+
auto_create_table: true, table: users, read_timeout_sec: 360000, open_timeout_sec: 360000,
|
114
|
+
send_timeout_sec: 360000, mode: append}
|
115
|
+
|
116
|
+
|
117
|
+
※プレビュー
|
118
|
+
|
119
|
+
C:\Users\yazaki\Desktop\embulk>embulk-0.9.8.bat preview config_bq_incremental.yml
|
120
|
+
2018-12-05 11:27:22.618 +0900: Embulk v0.9.8
|
121
|
+
2018-12-05 11:27:23.304 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
|
122
|
+
2018-12-05 11:27:25.457 +0900 [INFO] (main): Gem's home and path are set by default: "C:\Users\yazaki.embulk\lib\gems"
|
123
|
+
2018-12-05 11:27:26.190 +0900 [INFO] (main): Started Embulk v0.9.8
|
124
|
+
2018-12-05 11:27:26.268 +0900 [INFO] (0001:preview): Loaded plugin embulk-input-oracle (0.9.3)
|
125
|
+
2018-12-05 11:27:26.299 +0900 [INFO] (0001:preview): Connecting to jdbc:oracle:thin:@A0430-02396:1521:ORCL options {oracle.jdbc.ReadTimeout=1800000, user=system, password=***, oracle.net.CONNECT_TIMEOUT=300000}
|
126
|
+
2018-12-05 11:27:26.674 +0900 [INFO] (0001:preview): Using JDBC Driver 12.1.0.2.0
|
127
|
+
2018-12-05 11:27:26.783 +0900 [INFO] (0001:preview): Connecting to jdbc:oracle:thin:@A0430-02396:1521:ORCL options {oracle.jdbc.ReadTimeout=1800000, user=system, password=***, oracle.net.CONNECT_TIMEOUT=300000}
|
128
|
+
2018-12-05 11:27:26.845 +0900 [INFO] (0001:preview): SQL: SELECT * FROM "INPUT_TEST" ORDER BY "ID"
|
129
|
+
2018-12-05 11:27:26.861 +0900 [INFO] (0001:preview): > 0.02 seconds
|
130
|
+
+-----------+------------+------------+---------------+-------------------------+-------------------------+-----------------------------+-----------------------------+
|
131
|
+
| ID:string | NUM:string | STR:string | VARSTR:string | DT:timestamp | TIME0:timestamp | TIME6:timestamp | TIME9:timestamp |
|
132
|
+
+-----------+------------+------------+---------------+-------------------------+-------------------------+-----------------------------+-----------------------------+
|
133
|
+
| 179510 | 81596323 | OYYSHALN | PMTVRDKR | 2018-12-04 08:57:26 UTC | 2018-12-04 08:57:27 UTC | 2018-12-04 08:57:26.922 UTC | 2018-12-04 08:57:26.922 UTC |
|
134
|
+
| 179511 | 45967077 | EDMHMKSA | HCQMWTDO | 2018-12-04 08:57:26 UTC | 2018-12-04 08:57:27 UTC | 2018-12-04 08:57:26.969 UTC | 2018-12-04 08:57:26.969 UTC |
|
135
|
+
| 179512 | 45306398 | ZTGYAJZG | JGMEIMTI | 2018-12-04 08:57:26 UTC | 2018-12-04 08:57:27 UTC | 2018-12-04 08:57:26.969 UTC | 2018-12-04 08:57:26.969 UTC |
|
136
|
+
|
137
|
+
※実行コマンド
|
138
|
+
C:\Users\yazaki\Desktop\embulk>embulk-0.9.8.bat run config_bq_incremental.yml -c diff.yml >> C:\Users\yazaki\Desktop\embulk\log\ORA_to_BQ_log004.txt
|
139
|
+
|
140
|
+
|
141
|
+
※実行後のエラー
|
142
|
+
org.embulk.exec.PartialExecutionException: org.embulk.spi.DataException: Column type 'string' set at incremental_columns option is not supported
|
143
|
+
at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:339)
|
144
|
+
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:565)
|
145
|
+
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:34)
|
146
|
+
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:352)
|
147
|
+
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:349)
|
148
|
+
at org.embulk.spi.Exec.doWith(Exec.java:22)
|
149
|
+
at org.embulk.exec.BulkLoader.run(BulkLoader.java:349)
|
150
|
+
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:161)
|
151
|
+
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:292)
|
152
|
+
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:156)
|
153
|
+
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:436)
|
154
|
+
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:91)
|
155
|
+
at org.embulk.cli.Main.main(Main.java:26)
|
156
|
+
Caused by: org.embulk.spi.DataException: Column type 'string' set at incremental_columns option is not supported
|
157
|
+
at org.embulk.input.jdbc.getter.AbstractColumnGetter.encodeToJson(org/embulk/input/jdbc/getter/AbstractColumnGetter.java:92)
|
158
|
+
at org.embulk.input.jdbc.AbstractJdbcInputPlugin$LastRecordStore.accept(org/embulk/input/jdbc/AbstractJdbcInputPlugin.java:441)
|
159
|
+
at org.embulk.input.jdbc.AbstractJdbcInputPlugin.run(org/embulk/input/jdbc/AbstractJdbcInputPlugin.java:490)
|
160
|
+
at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor.runInputTask(org/embulk/exec/LocalExecutorPlugin.java:271)
|
161
|
+
at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor.access$000(org/embulk/exec/LocalExecutorPlugin.java:196)
|
162
|
+
at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor$1.call(org/embulk/exec/LocalExecutorPlugin.java:235)
|
163
|
+
at org.embulk.exec.LocalExecutorPlugin$ScatterExecutor$1.call(org/embulk/exec/LocalExecutorPlugin.java:232)
|
164
|
+
at java.util.concurrent.FutureTask.run(java/util/concurrent/FutureTask)
|
165
|
+
at java.util.concurrent.ThreadPoolExecutor.runWorker(java/util/concurrent/ThreadPoolExecutor)
|
166
|
+
at java.util.concurrent.ThreadPoolExecutor$Worker.run(java/util/concurrent/ThreadPoolExecutor$Worker)
|
167
|
+
at java.lang.Thread.run(java/lang/Thread)
|
168
|
+
|
169
|
+
Error: org.embulk.spi.DataException: Column type 'string' set at incremental_columns option is not supported
|
170
|
+
|
92
171
|
```
|
2
質問更新
title
CHANGED
File without changes
|
body
CHANGED
@@ -8,49 +8,68 @@
|
|
8
8
|
|
9
9
|
環境:Windows7、OracleDB 12c、embulk 0.9.8
|
10
10
|
|
11
|
+
※2018/12/05更新
|
12
|
+
```ここに言語を入力
|
11
|
-
※
|
13
|
+
※Oracleテーブル定義
|
12
|
-
```
|
13
|
-
in: {type: oracle, driver_path: 'C:\Users\xxxx\Desktop\embulkDB\product\12.2.0\dbhome_1\sqldeveloper\jdbc\lib\ojdbc7.jar',
|
14
|
-
host: A0430-02396, user: system, password: xxxx, database: xxx, table: xxxx, incremental: true}
|
15
|
-
out: {type: bigquery, auth_method: json_key, json_keyfile: 'C:\Users\xxxx\Desktop\embulk\My
|
16
|
-
First Project-6041ffecfba7.json', project: xxxx, dataset: xxxx,
|
17
|
-
auto_create_table: true, table: users, incremental: true}
|
18
|
-
```
|
19
14
|
|
20
|
-
|
21
|
-
|
22
|
-
|
23
|
-
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
31
|
-
|
32
|
-
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
39
|
-
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
53
|
-
|
15
|
+
----------------- -------- ---
|
16
|
+
|
17
|
+
ID NOT NULL NUMBER(8)
|
18
|
+
NUM NUMBER(12,2
|
19
|
+
STR CHAR(8)
|
20
|
+
VARSTR VARCHAR2(8)
|
21
|
+
DT DATE
|
22
|
+
TIME0 TIMESTAMP(0
|
23
|
+
TIME6 TIMESTAMP(6
|
24
|
+
TIME9 TIMESTAMP(9
|
25
|
+
|
26
|
+
|
27
|
+
※config_bq.yml.liquid
|
28
|
+
|
29
|
+
in:
|
30
|
+
type: oracle
|
31
|
+
path_prefix: null
|
32
|
+
driver_path: C:\Users\yazaki\Desktop\embulkDB\product\12.2.0\dbhome_1\sqldeveloper\jdbc\lib\ojdbc7.jar
|
33
|
+
host: A0430-02396
|
34
|
+
user: system
|
35
|
+
password: yazaki
|
36
|
+
database: ORCL
|
37
|
+
table: INPUT_TEST
|
38
|
+
incremental: true
|
39
|
+
incremental_columns: [id]
|
40
|
+
skip_header_lines: 0
|
41
|
+
allow_extra_columns: false
|
42
|
+
allow_optional_columns: true
|
43
|
+
columns:
|
44
|
+
- {name: ID, type: string}
|
45
|
+
- {name: NUM, type: string}
|
46
|
+
- {name: STR, type: string}
|
47
|
+
- {name: VARSTR, type: string}
|
48
|
+
- {name: DT, type: string}
|
49
|
+
- {name: TIME0, type: timestamp, format: '%Y/%m/%d'}
|
50
|
+
- {name: TIME6, type: timestamp, format: '%Y/%m/%d'}
|
51
|
+
- {name: TIME9, type: timestamp, format: '%Y/%m/%d'}
|
52
|
+
out: {type: bigquery, auth_method: json_key, json_keyfile: 'C:\Users\yazaki\Desktop\embulk\My
|
53
|
+
First Project-6041ffecfba7.json', project: fluid-emissary-216806, dataset: embulk_test,
|
54
|
+
auto_create_table: true, table: users, read_timeout_sec: 360000, open_timeout_sec: 360000,
|
55
|
+
send_timeout_sec: 360000, mode: append}
|
56
|
+
|
57
|
+
|
58
|
+
※実行コマンド
|
59
|
+
embulk-0.9.8.bat run config_bq.yml.liquid -c diff.yml >> C:\Users\yazaki\Desktop\embulk\log\ORA_to_BQ_log003.txt
|
60
|
+
|
61
|
+
|
62
|
+
※実行後のエラー
|
63
|
+
org.embulk.exec.PartialExecutionException: org.embulk.spi.DataException: Column type 'double' set at incremental_columns option is not supported
|
64
|
+
at org.embulk.exec.BulkLoader$LoaderState.buildPartialExecuteException(BulkLoader.java:339)
|
65
|
+
at org.embulk.exec.BulkLoader.doRun(BulkLoader.java:565)
|
66
|
+
at org.embulk.exec.BulkLoader.access$000(BulkLoader.java:34)
|
67
|
+
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:352)
|
68
|
+
at org.embulk.exec.BulkLoader$1.run(BulkLoader.java:349)
|
69
|
+
at org.embulk.spi.Exec.doWith(Exec.java:22)
|
70
|
+
at org.embulk.exec.BulkLoader.run(BulkLoader.java:349)
|
71
|
+
at org.embulk.EmbulkEmbed.run(EmbulkEmbed.java:161)
|
72
|
+
at org.embulk.EmbulkRunner.runInternal(EmbulkRunner.java:292)
|
54
73
|
at org.embulk.EmbulkRunner.run(EmbulkRunner.java:156)
|
55
74
|
at org.embulk.cli.EmbulkRun.runSubcommand(EmbulkRun.java:436)
|
56
75
|
at org.embulk.cli.EmbulkRun.run(EmbulkRun.java:91)
|
@@ -70,60 +89,4 @@
|
|
70
89
|
|
71
90
|
Error: org.embulk.spi.DataException: Column type 'double' set at incremental_columns option is not supported
|
72
91
|
|
73
|
-
C:\Users\xxxx\Desktop\embulk>
|
74
|
-
|
75
|
-
```
|
76
|
-
|
77
|
-
状況更新
|
78
|
-
```ここに言語を入力
|
79
|
-
|
80
|
-
|
81
|
-
■Oracleテーブル定義
|
82
|
-
|
83
|
-
SQL> desc INPUT_TEST
|
84
|
-
名前 NULL? 型
|
85
|
-
----------------------------------------- -------- ----------------------------
|
86
|
-
ID NOT NULL NUMBER(8)
|
87
|
-
NUM NUMBER(12,2)
|
88
|
-
STR CHAR(8)
|
89
|
-
VARSTR VARCHAR2(8)
|
90
|
-
DT DATE
|
91
|
-
TIME0 TIMESTAMP(0)
|
92
|
-
TIME6 TIMESTAMP(6)
|
93
|
-
TIME9 TIMESTAMP(9)
|
94
|
-
|
95
|
-
SQL>
|
96
|
-
|
97
|
-
|
98
|
-
■Embulk定義
|
99
|
-
in: {type: oracle, driver_path: 'C:\Users\yazaki\Desktop\embulkDB\product\12.2.0\dbhome_1\sqldeveloper\jdbc\lib\ojdbc7.jar',
|
100
|
-
host: A0430-02396, user: system, password: yazaki, database: ORCL, table: INPUT_TEST, incremental: true, incremental_columns: id}
|
101
|
-
out: {type: bigquery, auth_method: json_key, json_keyfile: 'C:\Users\yazaki\Desktop\embulk\My
|
102
|
-
First Project-6041ffecfba7.json', project: fluid-emissary-216806, dataset: embulk_test,
|
103
|
-
auto_create_table: true, table: users, read_timeout_sec: 360000, open_timeout_sec: 360000, send_timeout_sec: 360000, mode: insert}
|
104
|
-
|
105
|
-
■Embulk実行
|
106
|
-
|
107
|
-
dos cmd > C:\Users\yazaki\Desktop\embulk\embulk-0.9.8.bat run C:\Users\yazaki\Desktop\embulk\config_bq.yml -c C:\Users\yazaki\Desktop\embulk\diff.yml >> C:\Users\yazaki\Desktop\embulk\ORA_to_BQ_log002
|
108
|
-
|
109
|
-
|
110
|
-
■ログ
|
111
|
-
PS C:\Windows\System32\WindowsPowerShell\v1.0> cat "C:\Users\yazaki\Desktop\embulk\ORA_to_BQ_log002" -wait -tail 10
|
112
|
-
2018-12-04 17:57:27.580 +0900: Embulk v0.9.8
|
113
|
-
2018-12-04 17:57:29.234 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
|
114
|
-
2018-12-04 17:57:34.704 +0900 [INFO] (main): Gem's home and path are set by default: "C:\Users\yazaki.embulk\lib\gems"
|
115
|
-
2018-12-04 17:57:36.763 +0900 [INFO] (main): Started Embulk v0.9.8
|
116
|
-
2018-12-04 17:57:36.919 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-oracle (0.9.3)
|
117
|
-
2018-12-04 17:57:44.440 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.4.9)
|
118
|
-
2018-12-04 17:57:45.410 +0900: Embulk v0.9.8
|
119
|
-
2018-12-04 17:57:47.110 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
|
120
|
-
2018-12-04 17:57:52.525 +0900 [INFO] (main): Gem's home and path are set by default: "C:\Users\yazaki.embulk\lib\gems"
|
121
|
-
2018-12-04 17:57:54.569 +0900 [INFO] (main): Started Embulk v0.9.8
|
122
|
-
2018-12-04 17:57:54.819 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-oracle (0.9.3)
|
123
|
-
2018-12-04 17:58:02.505 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.4.9)
|
124
|
-
2018-12-04 17:58:03.163 +0900: Embulk v0.9.8
|
125
|
-
2018-12-04 17:58:04.333 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
|
126
|
-
2018-12-04 17:58:10.153 +0900 [INFO] (main): Gem's home and path are set by default: "C:\Users\yazaki.embulk\lib\gems"
|
127
|
-
2018-12-04 17:58:12.368 +0900 [INFO] (main): Started Embulk v0.9.8
|
128
|
-
2018-12-04 17:58:12.555 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-oracle (0.9.3)
|
129
92
|
```
|
1
質問内容更新
title
CHANGED
File without changes
|
body
CHANGED
@@ -72,4 +72,58 @@
|
|
72
72
|
|
73
73
|
C:\Users\xxxx\Desktop\embulk>
|
74
74
|
|
75
|
+
```
|
76
|
+
|
77
|
+
状況更新
|
78
|
+
```ここに言語を入力
|
79
|
+
|
80
|
+
|
81
|
+
■Oracleテーブル定義
|
82
|
+
|
83
|
+
SQL> desc INPUT_TEST
|
84
|
+
名前 NULL? 型
|
85
|
+
----------------------------------------- -------- ----------------------------
|
86
|
+
ID NOT NULL NUMBER(8)
|
87
|
+
NUM NUMBER(12,2)
|
88
|
+
STR CHAR(8)
|
89
|
+
VARSTR VARCHAR2(8)
|
90
|
+
DT DATE
|
91
|
+
TIME0 TIMESTAMP(0)
|
92
|
+
TIME6 TIMESTAMP(6)
|
93
|
+
TIME9 TIMESTAMP(9)
|
94
|
+
|
95
|
+
SQL>
|
96
|
+
|
97
|
+
|
98
|
+
■Embulk定義
|
99
|
+
in: {type: oracle, driver_path: 'C:\Users\yazaki\Desktop\embulkDB\product\12.2.0\dbhome_1\sqldeveloper\jdbc\lib\ojdbc7.jar',
|
100
|
+
host: A0430-02396, user: system, password: yazaki, database: ORCL, table: INPUT_TEST, incremental: true, incremental_columns: id}
|
101
|
+
out: {type: bigquery, auth_method: json_key, json_keyfile: 'C:\Users\yazaki\Desktop\embulk\My
|
102
|
+
First Project-6041ffecfba7.json', project: fluid-emissary-216806, dataset: embulk_test,
|
103
|
+
auto_create_table: true, table: users, read_timeout_sec: 360000, open_timeout_sec: 360000, send_timeout_sec: 360000, mode: insert}
|
104
|
+
|
105
|
+
■Embulk実行
|
106
|
+
|
107
|
+
dos cmd > C:\Users\yazaki\Desktop\embulk\embulk-0.9.8.bat run C:\Users\yazaki\Desktop\embulk\config_bq.yml -c C:\Users\yazaki\Desktop\embulk\diff.yml >> C:\Users\yazaki\Desktop\embulk\ORA_to_BQ_log002
|
108
|
+
|
109
|
+
|
110
|
+
■ログ
|
111
|
+
PS C:\Windows\System32\WindowsPowerShell\v1.0> cat "C:\Users\yazaki\Desktop\embulk\ORA_to_BQ_log002" -wait -tail 10
|
112
|
+
2018-12-04 17:57:27.580 +0900: Embulk v0.9.8
|
113
|
+
2018-12-04 17:57:29.234 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
|
114
|
+
2018-12-04 17:57:34.704 +0900 [INFO] (main): Gem's home and path are set by default: "C:\Users\yazaki.embulk\lib\gems"
|
115
|
+
2018-12-04 17:57:36.763 +0900 [INFO] (main): Started Embulk v0.9.8
|
116
|
+
2018-12-04 17:57:36.919 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-oracle (0.9.3)
|
117
|
+
2018-12-04 17:57:44.440 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.4.9)
|
118
|
+
2018-12-04 17:57:45.410 +0900: Embulk v0.9.8
|
119
|
+
2018-12-04 17:57:47.110 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
|
120
|
+
2018-12-04 17:57:52.525 +0900 [INFO] (main): Gem's home and path are set by default: "C:\Users\yazaki.embulk\lib\gems"
|
121
|
+
2018-12-04 17:57:54.569 +0900 [INFO] (main): Started Embulk v0.9.8
|
122
|
+
2018-12-04 17:57:54.819 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-oracle (0.9.3)
|
123
|
+
2018-12-04 17:58:02.505 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.4.9)
|
124
|
+
2018-12-04 17:58:03.163 +0900: Embulk v0.9.8
|
125
|
+
2018-12-04 17:58:04.333 +0900 [WARN] (main): DEPRECATION: JRuby org.jruby.embed.ScriptingContainer is directly injected.
|
126
|
+
2018-12-04 17:58:10.153 +0900 [INFO] (main): Gem's home and path are set by default: "C:\Users\yazaki.embulk\lib\gems"
|
127
|
+
2018-12-04 17:58:12.368 +0900 [INFO] (main): Started Embulk v0.9.8
|
128
|
+
2018-12-04 17:58:12.555 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-oracle (0.9.3)
|
75
129
|
```
|