質問編集履歴
1
エラーとなった箇所を確認できたので追記
test
CHANGED
|
File without changes
|
test
CHANGED
|
@@ -2,27 +2,19 @@
|
|
|
2
2
|
|
|
3
3
|
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
AWSのDataPipelineを使用してDynamoDBにデータをインポートしたいのですが、
|
|
6
6
|
|
|
7
|
-
AWSのDataPipelineを使用してDynamoDBにデータをインポート
|
|
8
|
-
|
|
9
|
-
|
|
7
|
+
エラーとなってしまいます。
|
|
10
|
-
|
|
11
|
-
どうしてエラーとなってしまうのか、またどこのログを確認すれば
|
|
12
|
-
|
|
13
|
-
原因調査できるのかが不明で手の打ちようがありません。
|
|
14
8
|
|
|
15
9
|
|
|
16
10
|
|
|
17
|
-
以下の公式の手順に沿ってやってい
|
|
11
|
+
以下の公式の手順に沿ってやっています。
|
|
18
12
|
|
|
19
13
|
[https://docs.aws.amazon.com/ja_jp/datapipeline/latest/DeveloperGuide/dp-importexport-ddb-part1.html](https://docs.aws.amazon.com/ja_jp/datapipeline/latest/DeveloperGuide/dp-importexport-ddb-part1.html)
|
|
20
14
|
|
|
21
15
|
|
|
22
16
|
|
|
23
|
-
ログもどこを見ればいいのか分からないので、何が原因でエラーになっているのかも
|
|
24
|
-
|
|
25
|
-
分かりません。どこから手を付けていいかもわからない状況です。
|
|
17
|
+
何が原因でエラーになっているのか分かりません。どこから手を付けていいかもわからない状況です。
|
|
26
18
|
|
|
27
19
|
すいませんが何とかデータインポートできるようにお力をお貸しいただけないでしょうか。
|
|
28
20
|
|
|
@@ -32,13 +24,27 @@
|
|
|
32
24
|
|
|
33
25
|
|
|
34
26
|
|
|
35
|
-
手順通り進めてDataPipelineを起動しました。EC2のインスタンスとEMRは動いていますが
|
|
27
|
+
~~手順通り進めてDataPipelineを起動しました。EC2のインスタンスとEMRは動いていますが
|
|
36
28
|
|
|
37
|
-
しばらくすると、以下の画像の様に「CANCELED」となってしまい、異常終了してい
|
|
29
|
+
しばらくすると、以下の画像の様に「CANCELED」となってしまい、異常終了しています。
|
|
30
|
+
|
|
31
|
+
~~
|
|
32
|
+
|
|
33
|
+
]
|
|
38
34
|
|
|
39
35
|
|
|
40
36
|
|
|
37
|
+
以下の通り「Terminate After」を「10min」に変更してクラスタは起動できるようになりました。
|
|
38
|
+
|
|
39
|
+
|
|
40
|
+
|
|
41
|
+
|
|
42
|
+
|
|
43
|
+
|
|
44
|
+
|
|
45
|
+
しかし、クラスターでの処理時にエラーとなります。
|
|
46
|
+
|
|
41
|
-
|
|
42
48
|
|
|
43
49
|
|
|
44
50
|
|
|
@@ -46,16 +52,102 @@
|
|
|
46
52
|
|
|
47
53
|
|
|
48
54
|
|
|
49
|
-
|
|
55
|
+
データフォーマットについては[ここ](https://docs.aws.amazon.com/ja_jp/amazondynamodb/latest/developerguide/SampleData.LoadData.html)を参考に以下の通りにCSVを作成いたしました。
|
|
50
|
-
|
|
51
|
-
同じくCANCELEDとなってしまいました。
|
|
52
56
|
|
|
53
57
|
|
|
54
58
|
|
|
55
|
-
### インポートしようとしているJSONデータ
|
|
59
|
+
#### インポートしようとしているJSONデータ
|
|
56
60
|
|
|
57
61
|
```item.json
|
|
58
62
|
|
|
59
63
|
{"item":[{"PutRequest":{"Item":{"item-category":{"S":"1"},"item-code1":{"S":"1-0001"},"item-code2":{"S":"1-0002"},"item-code3":{"S":"1-0003"},"item-code4":{"S":"1-0004"},"item-code5":{"S":"1-0005"},"item-code6":{"S":"1-0006"},"item-code7":{"S":"1-0007"},"item-code8":{"S":"1-0008"},"item-code9":{"S":"1-0009"},"item-code10":{"S":"1-0010"},"item-code11":{"S":"1-0011"},"item-code12":{"S":"1-0012"},"item-code13":{"S":"1-0013"},"item-code14":{"S":"1-0014"}}}},{"PutRequest":{"Item":{"item-category":{"S":"2"},"item-code1":{"S":"2-0001"},"item-code2":{"S":"2-0002"},"item-code3":{"S":"2-0003"},"item-code4":{"S":"2-0004"},"item-code5":{"S":"2-0005"},"item-code6":{"S":"2-0006"},"item-code7":{"S":"2-0007"},"item-code8":{"S":"2-0008"},"item-code10":{"S":"2-0010"}}}},{"PutRequest":{"Item":{"item-category":{"S":"3"},"item-code1":{"S":"3-0001"},"item-code2":{"S":"3-0002"},"item-code5":{"S":"3-0005"},"item-code6":{"S":"3-0006"},"item-code7":{"S":"3-0007"},"item-code8":{"S":"3-0008"},"item-code9":{"S":"3-0009"},"item-code10":{"S":"3-0010"},"item-code11":{"S":"3-0011"},"item-code12":{"S":"3-0012"}}}},{"PutRequest":{"Item":{"item-category":{"S":"4"},"item-code1":{"S":"4-0001"},"item-code2":{"S":"4-0002"},"item-code3":{"S":"4-0003"},"item-code4":{"S":"4-0004"},"item-code5":{"S":"4-0005"},"item-code6":{"S":"4-0006"},"item-code7":{"S":"4-0007"},"item-code8":{"S":"4-0008"},"item-code9":{"S":"4-0009"},"item-code10":{"S":"4-0010"}}}},{"PutRequest":{"Item":{"item-category":{"S":"5"},"item-code1":{"S":"5-0001"},"item-code7":{"S":"5-0007"},"item-code8":{"S":"5-0008"},"item-code9":{"S":"5-0009"},"item-code10":{"S":"5-0010"},"item-code11":{"S":"5-0011"},"item-code12":{"S":"5-0012"},"item-code14":{"S":"5-0014"}}}},{"PutRequest":{"Item":{"item-category":{"S":"6"},"item-code1":{"S":"6-0001"},"item-code2":{"S":"6-0002"},"item-code3":{"S":"6-0003"},"item-code4":{"S":"6-0004"},"item-code5":{"S":"6-0005"},"item-code14":{"S":"6-0014"}}}},{"PutRequest":{"Item":{"item-category":{"S":"7"},"item-code1":{"S":"7-0001"},"item-code3":{"S":"7-0003"},"item-code4":{"S":"7-0004"},"item-code6":{"S":"7-0006"},"item-code7":{"S":"7-0007"},"item-code9":{"S":"7-0009"},"item-code11":{"S":"7-0011"},"item-code12":{"S":"7-0012"},"item-code13":{"S":"7-0013"},"item-code14":{"S":"7-0014"}}}},{"PutRequest":{"Item":{"item-category":{"S":"8"},"item-code1":{"S":"8-0001"},"item-code2":{"S":"8-0002"},"item-code3":{"S":"8-0003"},"item-code4":{"S":"8-0004"},"item-code9":{"S":"8-0009"},"item-code10":{"S":"8-0010"}}}}]}
|
|
60
64
|
|
|
61
65
|
```
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
|
|
69
|
+
どうやら上記ファイルを取り込めずにクラスタで処理エラーとなってしまっていて、
|
|
70
|
+
|
|
71
|
+
後続処理を実行できないみたいです。DataPipelineの方で確認できたログを以下に貼り付けます。
|
|
72
|
+
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
#### データパイプラインで出力されているアクティビティログの内容
|
|
76
|
+
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
19 Nov 2018 07:22:05,483 [INFO] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.taskrunner.TaskPoller: Executing: amazonaws.datapipeline.activity.EmrActivity@566caa0d
|
|
80
|
+
|
|
81
|
+
19 Nov 2018 07:22:05,649 [INFO] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.activity.EmrActivity: EMR transform starting.
|
|
82
|
+
|
|
83
|
+
19 Nov 2018 07:22:05,658 [INFO] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.cluster.EmrClient: EMR client waiting for cluster to enter ready state for jobflow id 'j-39MDB0QJCVMNW'.
|
|
84
|
+
|
|
85
|
+
19 Nov 2018 07:22:05,658 [INFO] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.cluster.EmrClient: EMR client checking if cluster is ready for jobflow with id 'j-39MDB0QJCVMNW'.
|
|
86
|
+
|
|
87
|
+
19 Nov 2018 07:22:06,041 [INFO] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.cluster.EmrClient: EMR client reports that cluster with jobflow id 'j-39MDB0QJCVMNW' is ready.
|
|
88
|
+
|
|
89
|
+
19 Nov 2018 07:22:06,106 [INFO] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.cluster.EmrClient: EMR client adding steps with request '{JobFlowId: j-39MDB0QJCVMNW,Steps: [{Name: df-103676927OD2BO9ACKYE_@TableLoadActivity_2018-11-19T07:15:53_Attempt=1,ActionOnFailure: CONTINUE,HadoopJarStep: {Properties: [],Jar: s3://dynamodb-emr-ap-northeast-1/emr-ddb-storage-handler/2.1.0/emr-ddb-2.1.0.jar,Args: [org.apache.hadoop.dynamodb.tools.DynamoDbImport, s3://dynamodb-importdata/item-content.json, item-content, 0.25]}}]}'
|
|
90
|
+
|
|
91
|
+
19 Nov 2018 07:23:06,921 [WARN] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.cluster.EmrUtil: EMR job flow named 'df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53' with jobFlowId 'j-39MDB0QJCVMNW' is in status 'WAITING' because of the step 'df-103676927OD2BO9ACKYE_@TableLoadActivity_2018-11-19T07:15:53_Attempt=1' failures 'null'
|
|
92
|
+
|
|
93
|
+
19 Nov 2018 07:23:06,921 [INFO] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.cluster.EmrUtil: EMR job '@TableLoadActivity_2018-11-19T07:15:53_Attempt=1' with jobFlowId 'j-39MDB0QJCVMNW' is in status 'WAITING' and reason 'Cluster ready after last step completed.'. Step 'df-103676927OD2BO9ACKYE_@TableLoadActivity_2018-11-19T07:15:53_Attempt=1' is in status 'FAILED' with reason 'null'
|
|
94
|
+
|
|
95
|
+
19 Nov 2018 07:23:06,922 [INFO] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.cluster.EmrUtil: Collecting steps stderr logs for cluster with AMI 3.9.0
|
|
96
|
+
|
|
97
|
+
19 Nov 2018 07:23:06,937 [INFO] (TaskRunnerService-df-103676927OD2BO9ACKYE_@EmrClusterForLoad_2018-11-19T07:15:53-0) df-103676927OD2BO9ACKYE amazonaws.datapipeline.taskrunner.LogMessageUtil: Returning tail errorMsg : at org.apache.hadoop.mapreduce.JobSubmitter.writeOldSplits(JobSubmitter.java:520)
|
|
98
|
+
|
|
99
|
+
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:512)
|
|
100
|
+
|
|
101
|
+
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:394)
|
|
102
|
+
|
|
103
|
+
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
|
|
104
|
+
|
|
105
|
+
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
|
|
106
|
+
|
|
107
|
+
at java.security.AccessController.doPrivileged(Native Method)
|
|
108
|
+
|
|
109
|
+
at javax.security.auth.Subject.doAs(Subject.java:415)
|
|
110
|
+
|
|
111
|
+
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
|
|
112
|
+
|
|
113
|
+
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
|
|
114
|
+
|
|
115
|
+
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:562)
|
|
116
|
+
|
|
117
|
+
at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:557)
|
|
118
|
+
|
|
119
|
+
at java.security.AccessController.doPrivileged(Native Method)
|
|
120
|
+
|
|
121
|
+
at javax.security.auth.Subject.doAs(Subject.java:415)
|
|
122
|
+
|
|
123
|
+
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
|
|
124
|
+
|
|
125
|
+
at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:557)
|
|
126
|
+
|
|
127
|
+
at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:548)
|
|
128
|
+
|
|
129
|
+
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:833)
|
|
130
|
+
|
|
131
|
+
at org.apache.hadoop.dynamodb.tools.DynamoDbImport.run(DynamoDbImport.java:68)
|
|
132
|
+
|
|
133
|
+
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
|
|
134
|
+
|
|
135
|
+
at org.apache.hadoop.dynamodb.tools.DynamoDbImport.main(DynamoDbImport.java:30)
|
|
136
|
+
|
|
137
|
+
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
|
|
138
|
+
|
|
139
|
+
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
|
|
140
|
+
|
|
141
|
+
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
|
|
142
|
+
|
|
143
|
+
at java.lang.reflect.Method.invoke(Method.java:606)
|
|
144
|
+
|
|
145
|
+
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
|
|
146
|
+
|
|
147
|
+
~以下略~
|
|
148
|
+
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
|
|
152
|
+
|
|
153
|
+
どの箇所でエラーとなっていてどこを修正すればいいかご教示のほどよろしくお願いいたします。
|