Improve csvsql insert performance using a transaction
Created by: migonzalvar
I added a transaction in csvsql.py. Performance for big files improves a lot.
--- a/csvkit/utilities/csvsql.py
+++ b/csvkit/utilities/csvsql.py
@@ -56,8 +56,12 @@ class CSVSQL(CSVKitUtility):
insert = sql_table.insert()
headers = csv_table.headers()
+ conn = engine.connect()
+ trans = conn.begin()
for row in csv_table.to_rows():
- engine.execute(insert, [dict(zip(headers, row)), ])
+ conn.execute(insert, [dict(zip(headers, row)), ])
+ trans.commit()
+ conn.close()
# Writing to file
else:
AFAIK, every single SQLalchemy database backend includes transaction support.
Could this little patch be applied to the code? Do you prefer a pull request?