文字の抜き出しが適切にできない

Question

「それぞれのwordがある文章の中で、何回登場するのか？」のタスクをやろうとしてます。
textと言う文章が与えられて、その文章のwordを小文字にし、それぞれのuniqueなwordの数を数えようとしているのですが、
下記の出力のように、wordだけでなく、「”」も取り出してしまっています。

原因と解決策をご教示いただけると幸いでございます。
下記は、私が試していることです。

text = "As I was waiting, a man came out of a side room, and at a glance I was sure he must be Long John. His left leg was cut off close by the hip, and under the left shoulder he carried a crutch, which he managed with wonderful dexterity, hopping about upon it like a bird. He was very tall and strong, with a face as big as a ham—plain and pale, but intelligent and smiling. Indeed, he seemed in the most cheerful spirits, whistling as he moved about among the tables, with a merry word or a slap on the shoulder for the more favoured of his guests."

def count_words(text):
    """Count how many times each unique word occurs in text."""
    counts = dict()  # dictionary of { <word>: <count> } pairs to return
    # TODO: Convert to lowercase
    text = text.lower()
    # TODO: Split text into tokens (words), leaving out punctuation
    # (Hint: Use regex to split on non-alphanumeric characters)
    text = re.split(r"\W+", text)
    # TODO: Aggregate word counts using a dictionary
    for char in text:
        counts[char] = text.count(char)
    return counts

print(count_words(text))

出力

Answer 1

” ではありません。'' です。空の文字列です。
\W+で分割しているので、文末の 'guests.' が 'guests' と '' に別れているのでしょう。

---

\W+で分割するのではなく、\w+を抽出するようアプローチを替えてみては。

text = re.findall(r'\w+', text)

Answer 2

自分で集計せずに、 groupbuy というメソッドを使う方法もあります。

w.py

from itertools import groupby

text = "As I was waiting, a man came out of a side room, and at a glance I was sure he must be Long John. His left leg was cut off clos\
e by the hip, and under the left shoulder he carried a crutch, which he managed with wonderful dexterity, hopping about upon it like a \
bird. He was very tall and strong, with a face as big as a ham—plain and pale, but intelligent and smiling. Indeed, he seemed in the m\
ost cheerful spirits, whistling as he moved about among the tables, with a merry word or a slap on the shoulder for the more favoured o\
f his guests."

for key, group in groupby(sorted(text.lower().split()), key=lambda x: x):
    print(key, len(list(group)))

参考情報

• 意外と知られていない!? Python標準ライブラリの関数groupby

https://qiita.com/tag1216/items/9aecc6816371ee35215e

Answer 3

どうでしょうか

text = "As I was waiting, a man came out of a side room, and at a glance I was sure he must be Long John. His left leg was cut off close by the hip, and under the left shoulder he carried a crutch, which he managed with wonderful dexterity, hopping about upon it like a bird. He was very tall and strong, with a face as big as a ham—plain and pale, but intelligent and smiling. Indeed, he seemed in the most cheerful spirits, whistling as he moved about among the tables, with a merry word or a slap on the shoulder for the more favoured of his guests."

def count_words(text):
    """Count how many times each unique word occurs in text."""
    counts = dict() 
    
    words = text.split()
    text = [word.lower() for word in words]

    for char in text:
        counts[char] = text.count(char)
    return counts

print(count_words(text))