質問編集履歴
4
編集
test
CHANGED
|
File without changes
|
test
CHANGED
|
@@ -2,7 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
更新されたソースコードを続行してもからのファイルが誕生してしまう。
|
|
6
|
+
|
|
7
|
+
うまく読み込めていないのがどこなのかいまいちわからない。
|
|
6
8
|
|
|
7
9
|
|
|
8
10
|
|
3
修正
test
CHANGED
|
File without changes
|
test
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
空のファイルが出来上がってしまう。
|
|
6
6
|
|
|
7
7
|
|
|
8
8
|
|
2
プログラムの更新
test
CHANGED
|
File without changes
|
test
CHANGED
|
@@ -12,37 +12,13 @@
|
|
|
12
12
|
|
|
13
13
|
```
|
|
14
14
|
|
|
15
|
-
|
|
15
|
+
エラー自体は消えました
|
|
16
|
-
|
|
17
|
-
|
|
16
|
+
|
|
18
|
-
|
|
19
|
-
File "ngram.py", line 76, in <module>
|
|
20
|
-
|
|
21
|
-
ngram = ng.get(text, ngram)
|
|
22
|
-
|
|
23
|
-
File "ngram.py", line 20, in get
|
|
24
|
-
|
|
25
|
-
seq = self.tagger.parse(text.encode('utf-8')).split()
|
|
26
|
-
|
|
27
|
-
File "/Users/ryoya/.pyenv/versions/anaconda3-4.3.1/lib/python3.6/site-packages/MeCab.py", line 281, in parse
|
|
28
|
-
|
|
29
|
-
def parse(self, *args): return _MeCab.Tagger_parse(self, *args)
|
|
30
|
-
|
|
31
|
-
NotImplementedError: Wrong number or type of arguments for overloaded function 'Tagger_parse'.
|
|
32
|
-
|
|
33
|
-
Possible C/C++ prototypes are:
|
|
34
|
-
|
|
35
|
-
MeCab::Tagger::parse(MeCab::Model const &,MeCab::Lattice *)
|
|
36
|
-
|
|
37
|
-
MeCab::Tagger::parse(MeCab::Lattice *) const
|
|
38
|
-
|
|
39
|
-
MeCab::Tagger::parse(char const *)
|
|
40
|
-
|
|
41
|
-
```
|
|
17
|
+
```
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
18
|
+
|
|
19
|
+
|
|
20
|
+
|
|
45
|
-
###
|
|
21
|
+
### 更新されたソースコード
|
|
46
22
|
|
|
47
23
|
|
|
48
24
|
|
|
@@ -80,68 +56,68 @@
|
|
|
80
56
|
|
|
81
57
|
self.N = N
|
|
82
58
|
|
|
83
|
-
self.tagger = MeCab.Tagger("-O
|
|
59
|
+
self.tagger = MeCab.Tagger("-Owakati")
|
|
84
60
|
|
|
85
61
|
|
|
86
62
|
|
|
87
63
|
def get(self, text, ngram=None):
|
|
88
64
|
|
|
89
|
-
seq = self.tagger.parse(text
|
|
65
|
+
seq = self.tagger.parse(text)
|
|
66
|
+
|
|
67
|
+
|
|
68
|
+
|
|
69
|
+
if ngram is None:
|
|
70
|
+
|
|
71
|
+
ngram = [defaultdict(int) for x in range((self.N + 1))]
|
|
72
|
+
|
|
73
|
+
ngram[0] = None
|
|
90
74
|
|
|
91
75
|
|
|
92
76
|
|
|
77
|
+
for i in range(len(seq)):
|
|
78
|
+
|
|
79
|
+
for n in range(1, self.N + 1):
|
|
80
|
+
|
|
81
|
+
idx = i - n + 1 # check ngram is valid range
|
|
82
|
+
|
|
93
|
-
if
|
|
83
|
+
if idx >= 0:
|
|
84
|
+
|
|
94
|
-
|
|
85
|
+
key_words = []
|
|
86
|
+
|
|
95
|
-
|
|
87
|
+
for j in range(idx, i+1):
|
|
88
|
+
|
|
96
|
-
|
|
89
|
+
key_words.append(seq[j])
|
|
90
|
+
|
|
91
|
+
key = '_'.join(key_words)
|
|
92
|
+
|
|
97
|
-
ngram[
|
|
93
|
+
ngram[n][key] += 1
|
|
98
94
|
|
|
99
95
|
|
|
100
96
|
|
|
97
|
+
return ngram
|
|
98
|
+
|
|
99
|
+
|
|
100
|
+
|
|
101
|
+
|
|
102
|
+
|
|
101
|
-
|
|
103
|
+
class HTMLParser():
|
|
102
|
-
|
|
104
|
+
|
|
105
|
+
|
|
106
|
+
|
|
103
|
-
|
|
107
|
+
def get(self, url):
|
|
104
|
-
|
|
105
|
-
|
|
108
|
+
|
|
106
|
-
|
|
107
|
-
|
|
109
|
+
try:
|
|
108
|
-
|
|
109
|
-
|
|
110
|
+
|
|
110
|
-
|
|
111
|
-
for j in range(idx, i+1):
|
|
112
|
-
|
|
113
|
-
key_words.append(seq[j])
|
|
114
|
-
|
|
115
|
-
|
|
111
|
+
c = urllib.request.urlopen(url)
|
|
112
|
+
|
|
116
|
-
|
|
113
|
+
except:
|
|
114
|
+
|
|
115
|
+
print("Could not open %s" % url)
|
|
116
|
+
|
|
117
|
-
|
|
117
|
+
return ""
|
|
118
118
|
|
|
119
119
|
|
|
120
120
|
|
|
121
|
-
return ngram
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
class HTMLParser():
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
def get(self, url):
|
|
132
|
-
|
|
133
|
-
try:
|
|
134
|
-
|
|
135
|
-
c = urllib.request.urlopen(url)
|
|
136
|
-
|
|
137
|
-
except:
|
|
138
|
-
|
|
139
|
-
print("Could not open %s" % url)
|
|
140
|
-
|
|
141
|
-
return ""
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
121
|
soup = BeautifulSoup(c.read(), "lxml")
|
|
146
122
|
|
|
147
123
|
text = '\n'.join(self.__getNavigableStrings(soup))
|
|
@@ -219,3 +195,15 @@
|
|
|
219
195
|
|
|
220
196
|
|
|
221
197
|
```
|
|
198
|
+
|
|
199
|
+
|
|
200
|
+
|
|
201
|
+
### URLS.JSONのファイル
|
|
202
|
+
|
|
203
|
+
|
|
204
|
+
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
["https://news.yahoo.co.jp/profile/settings/", "https://headlines.yahoo.co.jp/hl?a=20180601-00000063-dal-ent", "https://news.yahoo.co.jp/feature", "https://news.yahoo.co.jp/pickup/6284612", "https://news.yahoo.co.jp/pickup/6284600", "https://news.yahoo.co.jp/ranking/access?ty=t", "https://news.yahoo.co.jp/polls/", "https://news.yahoo.co.jp/hl?c=c_spo", "https://news.yahoo.co.jp/hl?c=loc", "https://www.yahoo-help.jp/app/noscript", "https://headlines.yahoo.co.jp/hl?a=20180601-00000520-san-pol", "https://news.yahoo.co.jp/search/advanced", "https://news.yahoo.co.jp/pickup/6284614", "https://news.yahoo.co.jp/pickup/6284627", "https://news.yahoo.co.jp/photo", "http://news.yahoo.co.jp/", "https://news.yahoo.co.jp/profile/login", "https://news.yahoo.co.jp/ranking", "https://headlines.yahoo.co.jp/purchase/", "https://headlines.yahoo.co.jp/hl?a=20180601-00000002-wordleafs-socc", "https://news.yahoo.co.jp/byline/", "https://headlines.yahoo.co.jp/videonews/jnn?a=20180601-00000013-jnn-soci", "https://news.yahoo.co.jp/ranking/access?ty=v", "https://about.yahoo.co.jp/docs/info/terms/chapter1.html", "https://headlines.yahoo.co.jp/hl?a=20180601-00000010-jij-soci", "https://news.yahoo.co.jp/ranking/access?ty=z", "https://headlines.yahoo.co.jp/videonews/jnn?a=20180530-00000072-jnn-soci", "https://headlines.yahoo.co.jp/article?a=20180601-00542887-shincho-ent", "https://about.yahoo.co.jp/info/mediastatement/", "https://headlines.yahoo.co.jp/docs/copyright.html", "https://news.yahoo.co.jp/byline/kohyoungki/20180531-00085870/", "https://news.yahoo.co.jp/ranking/comment/rate?ty=t", "https://news.yahoo.co.jp/ranking/access?ty=b", "https://headlines.yahoo.co.jp/hl?a=20180601-00003575-tokaiv-l23", "https://headlines.yahoo.co.jp/hl?a=20180531-00010002-bfj-soci", "https://news.yahoo.co.jp/hl?c=c_ent", "https://headlines.yahoo.co.jp/hl?a=20180601-00000519-san-pol", "https://news.yahoo.co.jp/pickup/6284617", "https://news.yahoo.co.jp/hl?c=dom", "https://news.yahoo.co.jp/", "https://headlines.yahoo.co.jp/article?a=20180601-00223078-toyo-soci", "https://news.yahoo.co.jp/topics", "https://headlines.yahoo.co.jp/videonews/", "https://news.yahoo.co.jp/hl?c=c_sci", "https://news.yahoo.co.jp/list/", "https://headlines.yahoo.co.jp/videonews/jnn?a=20180531-00000058-jnn-soci", "https://headlines.yahoo.co.jp/hl?a=20180601-00003574-tokaiv-l23", "https://about.yahoo.co.jp/docs/info/terms/", "https://www.yahoo-help.jp/app/home/p/575/", "https://about.yahoo.co.jp/info/msiesp/", "https://news.yahoo.co.jp/hl?c=c_life", "https://news.yahoo.co.jp/flash", "https://headlines.yahoo.co.jp/hl?a=20180601-00226047-nksports-socc", "https://news.yahoo.co.jp/pickup/6284619", "https://headlines.yahoo.co.jp/docs/tokuteisho.html", "https://news.yahoo.co.jp/byline/oshimakazuto/20180531-00085867/", "https://news.yahoo.co.jp/hl?c=bus", "https://headlines.yahoo.co.jp/hl?a=20180531-00000001-withnews-sci", "https://headlines.yahoo.co.jp/hl?a=20180601-00010002-khbv-l04", "https://www.yahoo-help.jp/app/answers/detail/a_id/43880/p/533/", "https://news.yahoo.co.jp/byline/fuwaraizo/20180531-00085864/", "https://feedback.ms.yahoo.co.jp/voc/news-voc/input", "https://news.yahoo.co.jp/hl?c=c_int", "https://news.yahoo.co.jp/pickup/6284621", "https://news.yahoo.co.jp/zasshi", "https://headlines.yahoo.co.jp/article?a=20180601-00055904-gendaibiz-bus_all", "https://www.yahoo-help.jp/app/answers/detail/p/575/a_id/60137", "https://news.yahoo.co.jp/pickup/6284618"]
|
|
208
|
+
|
|
209
|
+
```
|
1
誤字
test
CHANGED
|
File without changes
|
test
CHANGED
|
@@ -46,7 +46,7 @@
|
|
|
46
46
|
|
|
47
47
|
|
|
48
48
|
|
|
49
|
-
```
|
|
49
|
+
```
|
|
50
50
|
|
|
51
51
|
# coding: utf-8
|
|
52
52
|
|