pythonでbashのdiffコマンドを実現したい

Question

概要

CSVファイルA/B間のファイル差分をpythonで作成したツールで標準画面出力する

背景

pythonの学習を始め、CSVファイルの操作でつまづいている。

前提

学習に使用してしている環境は以下

• python3.7
• Anaconda
  • Pandas
• Jupyter Note

郵便局のken_all.csvの形式を使用を想定して以下のようなデータを用意する

before.csv

mkdir /tmp/ken_all
cd /tmp/ken_all
vim before.txt

13101,"100  ","1000000","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲｶﾆｹｲｻｲｶﾞﾅｲﾊﾞｱｲ","東京都","千代田区","以下に掲載がない場合",0,0,0,0,0,0
13101,"102  ","1020072","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲｲﾀﾞﾊﾞｼ","東京都","千代田区","飯田橋",0,0,1,0,0,0
13101,"101  ","1010032","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,0

after.csv

vim after.csv

13101,"100  ","1000000","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲｶﾆｹｲｻｲｶﾞﾅｲﾊﾞｱｲ","東京都","千代田区","以下に掲載がない場合",0,0,0,0,0,0
13101,"102  ","1020082","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾁﾊﾞﾝﾁｮｳ","東京都","千代田区","一番町",0,0,0,0,0,0
13101,"101  ","1010035","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,5

diff

ファイルの差分を確認

diff before.txt after.txt

2,4c2,4
< 13101,"102  ","1020072","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲｲﾀﾞﾊﾞｼ","東京都","千代田区","飯田橋",0,0,1,0,0,0
< 13101,"101  ","1010032","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
< 13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,0
---
> 13101,"102  ","1020082","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾁﾊﾞﾝﾁｮｳ","東京都","千代田区","一番町",0,0,0,0,0,0
> 13101,"101  ","1010035","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
> 13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,5

ゴール

目指す出力形式は以下

python3 diff_csv.py

-13101,"102  ","1020072","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲｲﾀﾞﾊﾞｼ","東京都","千代田区","飯田橋",0,0,1,0,0,0
+13101,"102  ","1020082","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾁﾊﾞﾝﾁｮｳ","東京都","千代田区","一番町",0,0,0,0,0,0
-13101,"101  ","1010032","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
+13101,"101  ","1010035","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
-13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,0
+13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,5

備考

• 理想に近いものはgit diffの出力形式
  • この結果を半角カナの順にソートしてあるものをイメージしている

@@ -1,4 +1,4 @@
 13101,"100  ","1000000","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲｶﾆｹｲｻｲｶﾞﾅｲﾊﾞｱｲ","東京都","千代田区","以下に掲載がない場合",0,0,0,0,0,0
-13101,"102  ","1020072","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲｲﾀﾞﾊﾞｼ","東京都","千代田区","飯田橋",0,0,1,0,0,0
-13101,"101  ","1010032","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
-13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,0
+13101,"102  ","1020082","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾁﾊﾞﾝﾁｮｳ","東京都","千代田区","一番町",0,0,0,0,0,0
+13101,"101  ","1010035","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
+13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,5

手順

import pandas as pd
file1 = '/tmp/ken_all/before.csv'
file2 = '/tmp/ken_all/after.csv'
data1 = pd.read_csv(file1, encoding="shift_jis")
data2 = pd.read_csv(file2, encoding="shift_jis")

Answer 1

git diff コマンドが理想に近いのであれば、それを Python 経由で実行した結果を使えばいいのではないでしょうか？

import subprocess

# コマンドを実行し、結果を受け取る。
command = 'git diff before.csv after.csv'
ret = subprocess.run(command.split(), stdout=subprocess.PIPE)
output = result.stdout.decode()

# 受け取った結果を行ごとにリストにする。
lines = output.splitlines()

# 確認
for i, line in enumerate(lines):
    print(i, line)

0 diff --git a/before.csv b/after.csv
1 index 9e4b895..dc56adb 100644
2 --- a/before.csv
3 +++ b/after.csv
4 @@ -1,4 +1,4 @@
5  13101,"100  ","1000000","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲｶﾆｹｲｻｲｶﾞﾅｲﾊﾞｱｲ","東京都","千代田区","以下に掲載がない場合",0,0,0,0,0,0
6 -13101,"102  ","1020072","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲｲﾀﾞﾊﾞｼ","東京都","千代田区","飯田橋",0,0,1,0,0,0
7 -13101,"101  ","1010032","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
8 -13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,0
9 \ No newline at end of file
10 +13101,"102  ","1020082","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾁﾊﾞﾝﾁｮｳ","東京都","千代田区","一番町",0,0,0,0,0,0
11 +13101,"101  ","1010035","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｲﾜﾓﾄﾁｮｳ","東京都","千代田区","岩本町",0,0,1,0,0,0
12 +13101,"101  ","1010047","ﾄｳｷｮｳﾄ","ﾁﾖﾀﾞｸ","ｳﾁｶﾝﾀﾞ","東京都","千代田区","内神田",0,0,1,0,0,5
13 \ No newline at end of file

追記

pandas だけで同じような出力をするなら以下のようになります。

1. 一致しない行を抽出する。
2. 一致しない行を交互にマージする。

import csv
from io import StringIO

import pandas as pd

# CSV を読み込む
before = pd.read_csv('before.txt', header=None)
after = pd.read_csv('after.txt', header=None)

# 左結合
diff_before = before.merge(after, indicator=True, how='left').loc[lambda x : x['_merge'] != 'both']
diff_after = after.merge(before, indicator=True, how='left').loc[lambda x : x['_merge'] != 'both']

# マーカー挿入
diff_before.insert(0, 'marker', '-')
diff_after.insert(0, 'marker', '+')

# 交互に結合する。
diff = pd.concat([diff_before, diff_after]).sort_index(kind='merge')
# _merge 列を削除
diff.drop(columns=['_merge'], inplace=True)

# DataFrame を出力する。
ofs = StringIO()
diff.to_csv(ofs, index=False, header=False)
output = ofs.getvalue()

print(output)

-,13101,102,1020072,ﾄｳｷｮｳﾄ,ﾁﾖﾀﾞｸ,ｲｲﾀﾞﾊﾞｼ,東京都,千代田区,飯田橋,0,0,1,0,0,0
+,13101,102,1020082,ﾄｳｷｮｳﾄ,ﾁﾖﾀﾞｸ,ｲﾁﾊﾞﾝﾁｮｳ,東京都,千代田区,一番町,0,0,0,0,0,0
-,13101,101,1010032,ﾄｳｷｮｳﾄ,ﾁﾖﾀﾞｸ,ｲﾜﾓﾄﾁｮｳ,東京都,千代田区,岩本町,0,0,1,0,0,0
+,13101,101,1010035,ﾄｳｷｮｳﾄ,ﾁﾖﾀﾞｸ,ｲﾜﾓﾄﾁｮｳ,東京都,千代田区,岩本町,0,0,1,0,0,0
-,13101,101,1010047,ﾄｳｷｮｳﾄ,ﾁﾖﾀﾞｸ,ｳﾁｶﾝﾀﾞ,東京都,千代田区,内神田,0,0,1,0,0,0
+,13101,101,1010047,ﾄｳｷｮｳﾄ,ﾁﾖﾀﾞｸ,ｳﾁｶﾝﾀﾞ,東京都,千代田区,内神田,0,0,1,0,0,5

Comments

[wwbQzhMkhhgEmhU]

→ tiitoiさんへ
pandas利用の際の差異抽出は行数が違うだけでアウトですよ。
前提として、
・同じ行がない
・順番は気にしない
が必要ですが、左結合でmergeしたもののうち同一でないものを選択する方法を左右でやれば抽出できそうです。
https://stackoverflow.com/questions/48647534/python-pandas-find-difference-between-two-data-frames
まぁ、前提によりdiffの出力とは違ってしまいますが...

[tiitoi]

ご指摘ありがとうございます。確かに単純比較だと各行が1対1対応している前提になるので、よくないですね。
紹介いただいた方法で修正してみました。

[wwbQzhMkhhgEmhU]

動かしていませんが、目視でコード確認しました。大丈夫かと。
私が言っていいのか分かりませんが、対応ありがとうございます。

[IST]

お二人にベストアンサーを付けたいですが、
修正も含めてご対応を頂いたこともありtiitoi様にベストアンサーをつけさせて頂きたいと思います。
ありがとうございました！！

Answer 2

tiitoiさん
コメントありがとうございます。
この度は、pythonを利用してツールを作成することでpython/Pandasの学習がしたいと考えております。
そのため、シェルを実行する方法も素敵ですがどちらかというと、pythonで完結させたいです。

Comments

[tiitoi]

回答に対するコメントは回答のコメント欄をご活用ください。
git diff ライクな結果というのは理解しているのですが、最終的に求める出力というのを記載いただいた2つのCSVの例でよいので、手動で作成して質問欄に記載いただけますか？
あと「半角カナの順」というのは、どの列を半角カナの順にソートした結果について言ってますでしょうか？

[IST]

質問欄のゴールに記載している6行が目指している出力になります。
Pythonファイルを実行したら、
記載しているような出力が得られることがゴールとなります。
半角カナのソートについては、
変更のあった行が大きなブロックになってしまうと見づらくなってしまうので、変更前の行の直後に変更後の結果行が来る、という表示にできたらベストだと考えています。

[tiitoi]

pandas でやる方法を追記しました。