Python Virtual Environment
Most beginner developers have a hard time understanding virtual environments, why they are useful and how to use them. I am going to explain why you want to use them, and then cover how to use them. There are a couple ways to use virtual environments, all solve the same problems but are set up differently.
What is a virtual environment?
When you install a package with pip (or conda) you install that package in the current environment. If you do not have a virtual environment, that ends up being root. Installing packages in root is considered bad practice as different packages (and your own code) may require different versions of a package that may not work with other packages in other projects.
Packages you will use globally on your operating system are typically installed in root, anything that is specific to a project should be installed in a virtual environment. A virtual environment simulates a fresh install of python where you can install only the packages required for your project and freeze the version as needed. Freezing a version means you install a version of a module like 1.5 and do not upgrade to future versions prior to testing the impact. A virtual environment also keeps all your modules from impacting other projects.
One project may need requests 2.18.4 (the current latest version), but that version may cause problems (until a particular bug that affects your project is fixed). Using virtual environments you can install two different versions and keep both projects happy. Your code is only part of the problems, your dependencies can have dependencies.
Setting up and using a virtual environment.
Before explaining how to set up a virtual environment, I will give an overview of the process, then I will cover each step in detail.
Using a virtual environment
- Create Virtual Environment
- Activate Virtual Environment
- Install Packages Requirements
Creating a virtual environment
There are quite a few ways to set up a virtual environment. A few common ways are:
virtualenvcondapipenv
There are a few more but they are additive to the ones listed above.
virtualenv
virtualenv is the defacto standard for virtual environments. It is fairly easy to use but can be confusing to remember the syntax and until you have done it a lot of times you will need to look up the process. Normally I would recommend and teach you this method.
conda
If you are using Anaconda/Miniconda (I highly recommend you do!) then you have conda available to you. It is a package manager like pip but even more powerful. If you are using Anaconda I would typically recommend this over virtualenv, but this has changed with the final option available to us.
pipenv
This is the final option I am presenting to you, and it is the one I will cover in detail and recommend you use as well. Pipenv is a newer project that replaces virtualenv and makes the entire process easier and cleaner. It acts a bit like Nodejs's superiornpm package manager. pipenv combines pip and virtualenv into one easy to use tool. It also has some security benefits as it uses hashes, something you won't need to worry about usually.
You will most likely need to install pipenv as it is not usually included in most python distributions by default. This will be used as a systemwide package and will not be installed in a virtual environment.
Install pipenv
python3 -m pip install pipenv
Once pipenv is installed, creating and using environments is fun and easy.
Creating a virtual environment
Change to the directory of your project or create a new folder if you are starting a new project. From there you just use pipenv like you would use for pip. If it is the first time installing a package in that folder, a new Pipfile and Pipefile.lock file. These files will manage what packages and versions you have installed for your project (this folder and downward).
After you install the packages you need for your project using pipenv install [PACKAGE]. Once you have finished installing all the packages you need for your project you will need to activate your virtual environment. pipenv will remind you after each package is installed wiht the following message:
To activate this project's virtualenv, run the following:
$ pipenv shell
Just use pipenv shell to switch to your virtual environment. While using your virtual environment you will see the name of the virtual environment on the left of your command prompt.
That's it, you now have a virtual environment with just the packages you installed. The versions of all the packages is maintained by pipenv and will ensure no other project will impact this project.
You can use pipenv graph to list the packages you have installed in your virtual environment.
Exit virtual environment
When you are done with this project or want to switch virtual environments, you can just type in exit and you will be returned to a normal command prompt.
But wait! There's more!
One thing that pipenv brings to the table that Nodejs also does very well is dev dependencies. These are packages you only need during development but will not be installed during production.
When installing a package that should only be used for development (say for example a package that reloads a server when the code base changes or a test framework) you use the --dev parameter.
pipenv install pytest --dev
You don't want a test framework installed in production, this allows you to maintain a separate list of packages that will not be installed unless you use pipenv install --dev to install all de v dependencies. This is extremely useful when working on multiple machines or with multiple users.
This should give you everything you need to create and use virtual environments to create safe project environments during your python development. If you have any questions or need more information, check out the pipenv project page.
My Python Tips Series
- f-strings in Python 3.6
- Underscores in numeric literals
- A better interactive shell
- Secrets Module - New in 3.6
- PEP 8
- Slices
- Named Tuples
- Destructuring
- Counter
- Type Annotation
- Jupyter Notebooks
- Getting Help
Why you should vote me as witness
Witness & Administrator of four full nodes
My recent popular posts
STEEM, STEEM Power, Vests, and Steem Dollars. wtf is this shit?
The truth and lies about 25% curation, why what you know is FAKE NEWS
WTF is a hardware wallet, and why should you have one?
GINABOT - The Secret to your Sanity on Steemit
How to calculate post rewards
Use SSH all the time? Time for a big boy SSH Client
How to change your recovery account
How curation rewards work and how to be a kick ass curator
Markdown 101 - How to make kick ass posts on Steemit
Work ON your business, not in your business! - How to succeed as a small business
You are not entitled to an audience, you need to earn it!
How to properly setup SSH Key Authentication - If you are logging into your server with root, you are doing it wrong!
Building a Portable Game Console
@themarkymark hits the mark again :D nice posts, covers just the right amount of detail about virtual environments :-)
explaind very well virtual enivornment @themarkymark
That's very useful tip! I didn't know you can have virtual environments like that. It will be very useful for packages that have dependence of other packages of a particular version.
Very good post, few give a moment to guide those who do not handle this information well ...
nice post important informations
This is new to me. I Never came Python environment or the virtual environment concept you are talking about here. But hey, we learn everyday. Now I know.
Do you program in python?
It seems that virtual environments are becoming almost mandatory in more and more areas of delopment.
In web development we see more and more people using Virtual Machines or Docker.
I'm developing on Windows but a lot of production environments and build environments are running Linux.
And then it's just a super practical thing to be able to replicate a Linux environment on my machine.
It eliminates a lot of problems.
Similarly it's basically not worth it to try to run Ruby on Rails in a Windows environment.
But with a bit of setup. it's really not that difficult to setup a Linux environment in a container, so that you can keep using your favorite tools on Windows.
I'm not currently working on python but when/if I start I'll make sure to refrence your posts. Very thurough information and insturctions.
interesting
Hi @themarkymark An interesting work. Well-written, very well-documented. My son studying Computer Science in college.
Python applications will often use packages and modules that don’t come as part of the standard library. Applications will sometimes need a specific version of a library, because the application may require that a particular bug has been fixed or the application may be written using an obsolete version of the library’s interface.
This means it may not be possible for one Python installation to meet the requirements of every application. If application A needs version 1.0 of a particular module but application B needs version 2.0, then the requirements are in conflict and installing either version 1.0 or 2.0 will leave one application unable to run.
The solution for this problem is to create a virtual environment, a self-contained directory tree that contains a Python installation for a particular version of Python, plus a number of additional packages.
Different applications can then use different virtual environments. To resolve the earlier example of conflicting requirements, application A can have its own virtual environment with version 1.0 installed while application B has another virtual environment with version 2.0. If application B requires a library be upgraded to version 3.0, this will not affect application A’s environment.
I'm wishing you success and growth, and happiness.
A hug
Most deluxe work you are presinting here in your blog the great python application which have new modules and installment and many more packages yes it will work and definitely it will go for hihh note on application world it an interesting approaches for those who have difficult to find clusters of application on one package but python installment will solve this problem nicely you have shared a great idea i think it will go on top of top as soon as possible the way of its own valuable facts and figures
Cool article, but I don't agree with using pipenv. From what I understand pipenv is just a wrapper for people who can't use virtualenv. Virtualenv is pretty easy for me to use and straightforward , so I'll stick to that for now.
Here are some links that are related for anyone interested:
One of my favourite practices is to often cleanup my virtualenv and reinstall dependencies, especially in big projects with many dependencies. This helps me solve dependency issues,but can be a pain in the ass for development.
Create list of installed packages:
virt-env ) $ pip3 freeze > uninstall-requirements.txt virt-env ) $ pip3 uninstall -r uninstall-requirements.txt -y virt-env ) $ pip3 install -r requirements.txtPipenv is not a wrapper, there is virtualenvwrapper that you are likely thinking of.
Pipenv is a completely different project that works differently from virtualenv but does the same thing just better.
Ok fair enough. I work in Python everyday so I should have probably known that.
What makes it better than virtualenv?
Btw here are some old notes of mine about virtualenv, not sure if it is still relevant, but it was pretty interesting when I learned about this:
https://yoirtuts.com/index.php?title=Virtualenv_Correct_Usage
WIthout getting complex, there are a lot of advantages. I really recommend checking out the project page and within five minutes I will bet you will see the advantages.
It is not just an ease of use wrapper, it's a complete makeover on how virtual environments work on python and puts it in the same league as NodeJS's package manager.
Thanks. I'm actually sitting and working on a Django project right now. Gonna try it right now :)
Let me know what you think!
I tried installing it in a virtualenv and then I saw it installed virtualenv inside the virtualenv, which is like virtualenv inception. I'm going to try it again tomorrow. I think the change of workflow will take some time to adapt for me. I weirdly like uninstalling everything in my virtualenv and then installing from scratch from my requirements.txt, and the reason behind that is I work on some projects that have a huge list of dependencies(far too many for my liking) and often a package gets out of date or can't install anymore or a certain bug only creeps in with a specific version of a package. Are there any tools out there for converting your requirements.txt to the format that pipenv uses?
Why slowly, you can learn it in about 5 minutes, and benefit from it by the 6th. It's really a huge improvement over previous options.
You can find me on our Discord or many others.