There are two classes of Python users, those who use Python for management scripting and web development, and those who use Python for scientific computing. According to many, Python is winning the war against R in the field of machine learning. As machine learning becomes ever more important and popular, so is Python.

This article is for those just got into the Python world. For those experienced Python developers, you are excused to leave now. Python is extremely simple to start with, the traditional Hello World program takes only one line. Before no time, you would be writing classes, methods, system calls, and you really enjoy the convenience and power of the language. But quickly, you also realize that, there are many strange codes floating around. And you hear about lots of unacquainted jargons. You could feel a little puzzled, and a little confused about the big picture of the Python world.

That is where this article comes to help. I intend to draw a broad picture of the Python world, with pointers for you to explore in depth. At the end of the article, you won’t become an expert on anything with Python. But, hopefully, you will be satisfied in understanding the big picture, and know where to go for further information and to gain expertise.

Installation and environments

All software packages start with installations. As of the writing of this post, there are two major versions of Python that are prevalent, i.e. Python 2 and Python 3. Usually, Python 2 refers to Python with version 2.7.x, and Python 3 refers to Python with version 3.5+. Python 2 and Python 3 are incompatible on many occasions. Which one to choose? The simplified answer is to choose Python 3 for new projects and use whatever version your old project was using for the old projects.

Chances are, in the near future, you will need to work in both Python 2 and Python 3 worlds for different projects. So you install both. But then, how do you switch between the two? When you run python -m xxx command, how do you know which version is started? Do you need to manually set up your environment variables for switching? That would be too complicated and error prone, and you need tools. That is when Virtualenv and Anaconda come into play.

Virtualenv and Anaconda solve more problems than Python versions. Remember, for any real projects, you would need support from many libraries, right? Different projects usually rely on different versions of the same libraries, i.e. it is incorrect to assume all of your projects can always use the latest version of the same libraries. For different projects could be at different lifecycles. So the industry’s best practice is to isolate the python version + specific libraries versions into one bundle called: environments.

The difference between Virtualenv and Anaconda? Again, the simplest answer is

functionality wise. Wait, what is Pip? Be patient, I will talk about it. But before that, I’d like to show some codes:

On MacOS, with Virtualenv, commands to install and setup both Python 2 & 3 environments are:

brew install python

brew install python3

virtualenv env2

virtualenv --python=python3 env3

In the above codes, env2 and env3 and simply subfolders (that you can change to your favorite folder names) where the environments will be setup, i.e. all python binaries and libraries will be install and copied into those subfolders.

Environments need to be activated before invoking Python commands:

source env2/bin/activate

deactivate

source env3/bin/activate

deactivate

For Anaconda:

conda create --prefix ~/bio-env python=3.4

source activate ~/bio-env

source deactivate

In the above codes, you can change to the Python version to use Python 2.

Package installer

As promised, I will talk about Pip. As you would expect, any programmer needs libraries. In Python, libraries are called packages. Like in Java world, you pull and install packages from some central repositories called PyPI – the Python Package Index. Most likely you don’t have to install Pip. For Pip is installed when you install your Python environments.

Some sample pip commands for installing packages:

pip install SomePackage            # latest version

pip install SomePackage==1.0.4     # specific version

pip install 'SomePackage>=1.0.4'     # minimum version

pip install -r requirements.txt . # "Requirements files" are files containing a list of items to be installed

pip list # list installed packages

pip search "query" # search for packages

pip install --upgrade SomePackage # upgrade packages if needed

Another concept you need to understand is Python Wheels.

, by default, installs from source files involving compilation. This process could be demanding at times, especially when non-python libraries are involved. So the invention of the Python Wheels is to allow others to pre-built a package for you, so that you can quickly install without worrying about how to compile the package. Make sense, right?

pip install SomePackage-1.0-py2.py3-none-any.whl # installing from wheels

pip install wheel

pip wheel --wheel-dir=/local/wheels -r requirements.txt # To build wheels to a local directory

Remember Anaconda? It gets its own package installer tool. It is called Conda. Its usage patttern is similar to pip, but using a different central repository, the one from Continnum.

conda create --name snakes python=3 # create python 3 environment called snakes

conda info --envs # display the list of environments

source activate snowflakes # activate snowflakes environment

conda search beautiful-soup # search packages

conda install package_name # install a package

conda list # list installed packages

conda update package_name # update a package

conda update conda # update conda itself

conda update python # update to the latest 2.x or 3.x version of python

conda remove packge_name # uninstall a package

In addition to the central repository, Anaconda also supports “private” repositories hosted on anaconda.org. The “private” repositories are called channels. In install from a channel, you can use the -c option, e.g.

conda install -c pandas bottleneck

Don’t worry, with Anaconda, you can still use Pip.

I personally prefer Anaconda because it has more integrated support cross platforms and it has enough packages for scientific computing.

Your own modules and packages

With the installation and environments taken care of, you are ready to do some serious coding.

You write a python file with the .py as the extension. Congratulations, you’ve got your first module developed. Of course, you will want to write your own libraries for others to use. Well it is quite somple:

1. Create a folder named after your library name
2. Within that folder, put an empty __init__.py in it

Congratulations, you got your first package ready. Of course, it has no meat in it. For you have not done any real coding.

If you want some classes visible to others when they write

, you define the class in the __init__.py object. If you prefer other to write

or

, you keep the __init__.py empty and create

.

Don’t expect me to talk about how to write Python classes. It should be another of your Google searches.

Install your own packages

Time to share your work with others. Visit this page to equip yourself with the full knowledge. Basically you need to provide a set of meta information to inform the packager how to package your stuff. Briefly:

• setup.py – this is your main file containing a call of setup() from Setuptools package indicating package name and url etc.
• setup.cfg – an ini file that contains option defaults for setup.py commands
• MANIFEST.in – needed in certain cases where you need to package additional files such as resource files
• optional stuff – license file, author file, contributions file etc

The following commands will help you to have a sense how to package.

python setup.py sdist # create a source distribution

python setup.py bdist_wheel --universal # a universal (meaning pure python) wheel

If you need to upload your package into the PyPi repository to have a broader audience, you further need the twine tool.

Deploy your packages

For any serious project, you would need to deploy it to some servers and get it running. The Fabric package is here to make it easy. Python Fabric allows you to easily invoke shell scripts locally or remotely on the servers.

At the top of your project folder, you want to create a

which contains your deployment scripts. Some important imports at the top of that script:

from fabric.api import task, local, settings

from fabric.colors import green, red

from fabric.context_managers import prefix

Each method you define in the

becomes a subcommand as the

. For example, the following routine:

@task

def mytask():

  run("a command")

Sould allow you to invoke command from within the top folder of your project:

fab mytask

It is generally a good practice to have separate tasks for bootstrapping, deployment and testing. A tutorial is available here.

Python unit testing

We should never forget about tests whenever we are coding, right? Pytest is the tool to use for that purpose.

Welcome to REST

Here the REST means Representational state transfer. We often need to deliver a micro service out of Python. Then Flask is a good choice for that. It is RESTful, it supports templating and it is fully WSGI compliant.

Expand your horizon

• IPython – I just can’t stress enough how useful this tool is. It is a python prompt on steroids. It has completion, history, shell capabilities, and a lot more. Make sure that you take a look at it.
• NumPy – You do math or science? You can not live without it.
• SciPy – When we talk about NumPy then we have to talk about scipy. It is a library of algorithms and mathematical tools for python and has caused many scientists to switch from ruby to python.
• matplotlib – A numerical plotting library. It is very useful for any data scientist or any data analyzer.
• SymPy – SymPy can do algebraic evaluation, differentiation, expansion, complex numbers, etc. It is contained in a pure Python distribution.
• NLTK – Natural Language Toolkit – I realize most people won’t be using this one, but it’s generic enough. It is a very useful library if you want to manipulate strings. But it’s capacity is beyond that. Do check it out.
• SQLAlchemy – A database library. Many love it and many hate it. The choice is yours.
• BeautifulSoup – I know it’s slow but this xml and html parsing library is very useful for beginners.
• Requests – The most famous http library written by kenneth reitz. It’s a must have for every python developer.
• Cython – an extension language for the CPython runtime. It translates Python code to fast C code and supports calling external C and C++ code natively.
• Pandas – a library written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating numerical tables and time series. Pandas is free software released under the three-clause BSD license
• Statsmodels – a Python module that allows users to explore data, estimate statistical models, and perform statistical tests. An extensive list of descriptive statistics, statistical tests, plotting functions, and result statistics are available for different types of data and each estimator.
• scikit-learn – is an open source library for the Python. It features various classification, regression and clustering algorithms including support vector machines, logistic regression, naive Bayes, random forests, gradient boosting, k-means and DBSCAN, and is designed to interoperate with the Python numerical and scientific libraries NumPy and SciPy.
• Theano – uses NumPy-like syntax to optimize and evaluate mathematical expressions. What sets Theano apart is that it takes advantage of the computer’s GPU in order to make data-intensive calculations up to 100x faster than the CPU alone. Theano’s speed makes it especially valuable for deep learning and other computationally complex tasks.
• TensorFlow – another high-profile entrant into machine learning, developed by Google as an open-source successor to DistBelief, their previous framework for training neural networks. TensorFlow uses a system of multi-layered nodes that allow you to quickly set up, train, and deploy artificial neural networks with large datasets. It’s what allows Google to identify objects in photos or understand spoken words in its voice-recognition app.
• Scrapy – an aptly named library for creating spider bots to systematically crawl the web and extract structured data like prices, contact info, and URLs. Originally designed for web scraping, Scrapy can also extract data from APIs.
• Pattern – combines the functionality of Scrapy and NLTK in a massive library designed to serve as an out-of-the-box solution for web mining, NLP, machine learning, and network analysis. Its tools include a web crawler; APIs for Google, Twitter, and Wikipedia; and text-analysis algorithms like parse trees and sentiment analysis that can be performed with just a few lines of code.
• Seaborn – a popular visualization library that builds on matplotlib’s foundation. The first thing you’ll notice about Seaborn is that its default styles are much more sophisticated than matplotlib’s. Beyond that, Seaborn is a higher-level library, meaning it’s easier to generate certain kinds of plots, including heat maps, time series, and violin plots.
• Bokeh – another great visualization library is Bokeh, which is aimed at interactive visualizations. In contrast to the previous library, this one is independent of Matplotlib. The main focus of Bokeh, as we already mentioned, is interactivity and it makes its presentation via modern browsers in the style of Data-Driven Documents
• Keras – an open-source library for building Neural Networks at a high-level of the interface, and it is written in Python. It is minimalistic and straightforward with high-level of extensibility. It uses Theano or TensorFlow as its backends, but Microsoft makes its efforts now to integrate CNTK (Microsoft’s Cognitive Toolkit) as a new back-end.

Understanding all these, you are officially an expert!