fast_to_sql
is an improved way to upload pandas dataframes to Microsoft SQL Server.
fast_to_sql
takes advantage of pyodbc rather than SQLAlchemy. This allows for a much lighter weight import for writing pandas dataframes to sql server. It uses pyodbc's executemany
method with fast_executemany
set to True
, resulting in far superior run times when inserting data.
pip install fast_to_sql
- Written for Python 3.8+
- Requires pandas, pyodbc
from datetime import datetime
import pandas as pd
import pyodbc
from fast_to_sql import fast_to_sql as fts
# Test Dataframe for insertion
df = pd.DataFrame({
"Col1": [1, 2, 3],
"Col2": ["A", "B", "C"],
"Col3": [True, False, True],
"Col4": [datetime(2020,1,1),datetime(2020,1,2),datetime(2020,1,3)]
})
# Create a pyodbc connection
conn = pyodbc.connect(
"""
Driver={ODBC Driver 17 for SQL Server};
Server=localhost;
Database=my_database;
UID=my_user;
PWD=my_pass;
"""
)
# If a table is created, the generated sql is returned
create_statement = fts.fast_to_sql(df, "my_great_table", conn, if_exists="replace", custom={"Col1":"INT PRIMARY KEY"}, temp=False)
# Commit upload actions and close connection
conn.commit()
conn.close()
fts.fast_to_sql(df, name, conn, if_exists="append", custom=None, temp=False, copy=False)
df
: pandas DataFrame to uploadname
: String of desired name for the table in SQL serverconn
: A valid pyodbc connection objectif_exists
: Option for what to do if the specified table name already exists in the database. If the table does not exist a new one will be created. By default this option is set to 'append'- 'append': Appends the dataframe to the table if it already exists in SQL server.
- 'fail': Purposely raises a
FailError
if the table already exists in SQL server. - 'replace': Drops the old table with the specified name, and creates a new one. Be careful with this option, it will completely delete a table with the specified name in SQL server.
custom
: A dictionary object with one or more of the column names being uploaded as the key, and a valid SQL column definition as the value. The value must contain a type (INT
,FLOAT
,VARCHAR(500)
, etc.), and can optionally also include constraints (NOT NULL
,PRIMARY KEY
, etc.)- Examples:
{'ColumnName':'varchar(1000)'}
{'ColumnName2':'int primary key'}
- Examples:
temp
: EitherTrue
if creating a local sql server temporary table for the connection, orFalse
(default) if not.copy
: Defaults toFalse
. If set toTrue
, a copy of the dataframe will be made so column names of the original dataframe are not altered. Use this if you plan to continue to use the dataframe in your script after runningfast_to_sql
.