You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Describe the bug
I noticed that downloading and running LLMS such as llama2 and others are very slow. It takes lots of time in my local system to download the model before generating completions. Typically it is faster on Collab with quantization technique.
To Reproduce
Steps to reproduce the behavior:
Write any valid LLM-VM completions generation codes.
Select 'llama2' as 'big_model ' parameter.
Run or execute your code.
See error
Expected behavior
The whole process of downloading LLM models, shards, checkpoints,.. and completions processes should be faster . Perhaps everything should be done in 3-6 minutes. It takes average of 3-4 minutes in Collab.
Screenshots
Desktop (please complete the following information):
OS: Windows. Python 3.11.2
Version Windows 10 Pro
RAM Size: 16GB
Additional context
I am not running the code from Notebook, yet to try it on jupyter notebook. Rather I am running from command prompt that comes with PyCharm IDE.
The text was updated successfully, but these errors were encountered:
AfamO
changed the title
Downloading LLama2 and Other Models Running So Slow
Downloading LLama2 and Other LLMs Running So Slow
Jan 9, 2024
Describe the bug
I noticed that downloading and running LLMS such as llama2 and others are very slow. It takes lots of time in my local system to download the model before generating completions. Typically it is faster on Collab with quantization technique.
To Reproduce
Steps to reproduce the behavior:
Expected behavior
The whole process of downloading LLM models, shards, checkpoints,.. and completions processes should be faster . Perhaps everything should be done in 3-6 minutes. It takes average of 3-4 minutes in Collab.
Screenshots
![llama-running-slowly](https://private-user-images.githubusercontent.com/9824566/295220418-7ed3388d-3124-4624-b669-1cd3a2fe99dc.PNG?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MjU3NTgwMTMsIm5iZiI6MTcyNTc1NzcxMywicGF0aCI6Ii85ODI0NTY2LzI5NTIyMDQxOC03ZWQzMzg4ZC0zMTI0LTQ2MjQtYjY2OS0xY2QzYTJmZTk5ZGMuUE5HP1gtQW16LUFsZ29yaXRobT1BV1M0LUhNQUMtU0hBMjU2JlgtQW16LUNyZWRlbnRpYWw9QUtJQVZDT0RZTFNBNTNQUUs0WkElMkYyMDI0MDkwOCUyRnVzLWVhc3QtMSUyRnMzJTJGYXdzNF9yZXF1ZXN0JlgtQW16LURhdGU9MjAyNDA5MDhUMDEwODMzWiZYLUFtei1FeHBpcmVzPTMwMCZYLUFtei1TaWduYXR1cmU9YmMyMzViMjFlZmJhN2ZjNDFmNzdlMWE0NjhmNjY3MDdiMzlkMjkxYzdhYTcyNGNlZmZmMzAzYThkYTliMDZmMyZYLUFtei1TaWduZWRIZWFkZXJzPWhvc3QmYWN0b3JfaWQ9MCZrZXlfaWQ9MCZyZXBvX2lkPTAifQ.wHNHGLcr1BdK4c-gADQaFoH3ziXYP1RpePjglny1LB8)
Desktop (please complete the following information):
Additional context
I am not running the code from Notebook, yet to try it on jupyter notebook. Rather I am running from command prompt that comes with PyCharm IDE.
The text was updated successfully, but these errors were encountered: