Skip to content

Example of data import pipeline using Flow library

License

Notifications You must be signed in to change notification settings

anmarchenko/genstage_importer

 
 

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

genstage_importer

Demo project showing a concept of CSV import with GenStage/Flow data pipeline in Elixir. This is an illustration for a blogpost in Kloeckner's dev blog. CSV files to import are products.csv and orders.csv.

Setup

Requirements:

$ docker-compose up -d --build

Create a database:

$ docker-compose exec app mix ecto.setup

To run the sync:

$ docker-compose exec app iex -S mix
iex(1) > GenstageImporter.Pipeline.Product.import()

Generate test data

Requirements:

Edit script files/gen_csv.rb:

PRODUCTS_NUMBER = 1_000 # set number of products in CSV file here

Then run it:

$ ./files/gen_csv.rb

It will replace existing CSV files in ./files folder with new autogenerated data.

About

Example of data import pipeline using Flow library

Topics

Resources

License

Stars

Watchers

Forks

Languages

  • Elixir 87.4%
  • Ruby 7.1%
  • HTML 4.9%
  • Dockerfile 0.6%