Python quick reference

Resources

• for beginners without much programming background: Learn python the hard way
• a more systematic tutorial: python: tutorialspoint
• from R to Python: python for R

Contents

1. Special Characters
2. Variables and Operators
3. Functions
4. Input/Output
5. Control Flow
6. Data Structures & Classes
7. Modules
8. Python for data analysis
9. Selected topics
10. Other

Special characters

• #: add comments after #
• common escape sequences:
  \t, \n, \', \" ,\\ ...
• \: line continuation character
• ' ' and " " have the same effects
• use ; between statements to allow multiple statements in a single line
• True, False, None

Variables and operators

• basics:

    +,-,*,/ ...  # straight forward
  	
    %  # modulo operator
    // # floor division: take floor() after division
    a**b # exponential: a^b
  	
    # bitwise operator same as C
    A | B #Set union 
    A & B #Set intersection
    A & ~B	#Set subtraction 
    ALL_BITS ^ A or ~A 	#Set negation 
    A |= 1 << bit 	# Set 1 to bit 
    A &= ~(1 << bit) 	# Clear bit 
    (A & 1 << bit) != 0 	# Test bit 
    A&-A or A&~(A-1) or x^(x&(x-1)) 	#Extract last bit 
    A&(A-1) 	#Remove last bit 
    ~0 	#Get all 1-bits
  

  for a review of bitwise operations: bitwise operations review

• Logical variables: True, False
• Logical operators:
  not(),A or B, C and D
• check type of a variable:
  print type(variable) or print type(variable).__name__
• multiple assignment of variables:
  a,b,c = 1,2,'power'
• standard data types:
  • numbers: var1 = 1; var2 = 2.0 # var1 int, var2 double
del var1, var2 # remove variables
  • string
  • list
  • tuple
  • dictionary

• the dir() function:

    print dir() # all variables in the current space
    print dir(module_name) # all variable names in a module
  

• the unpack operator *:

    x= ['A','B','C']
    # pass A,B,C as separate arguments
    fun(*x) 
  

• global/local variables:

    global var # assign `var` to its global value in a local environment (eg. in a function)
    globals()  # gives a dict of global variables
    locals()   # gives a dict of local variables
  

Functions

• define a function:

    # a simple function example
    def my_fun(x,y):
        print "sum of x and y is ", x+y
        return (x,y,x+y) # or {'x':x,'z':x+y}
  	
    # use the function
    my_fun(2,3) 
    my_fun(x=2,y=3)	
  

  Note that you can return multiple things separated by , as a ordered list of items. Let out = my_fun(1,3). We can access values by out[0] (or out['x'] in the commented case)

• write a function with unknown number of arguments: *args and **kwargs

    # args handles all unnamed arguments and put in a tuple
    # kwargs handles all named arguments and put in a dictionary
    def myfun(*args,**kwargs): 
        print(args)
        print(kwargs)
    # examples	
    myfun(1,2,'x')
    #(1, 2, 'x')
    #{}	
    myfun(x=1,y=2,z='x')
    #()
    #{'x': 1, 'y': 2, 'z': 'x'}
  

• Pass by reference vs value
  All python parameters are passed by reference, i.e. when a parameter is used in a function, a new reference to the same object is created and used in the statements in the function body.

    # example function
    def change(mylist):
        mylist.append('2'); # change the object
        mylist += ['a','b']; # change the object
        mylist = mylist + ['c','d']; # does not change the object 
        # because at this line, a new object is created and mylist become a reference to that, while the old object remains
        return;
  

• variable length arguments in functions

    # example function
    def eg(arg1, *vartuple):
        print arg1
        for var in vartuple
            print var
        return
    # the argument followed by * holds the remaining parameters
  

• lambda functions (small anonymous functions)

    # lambda function example
    g = lambda x: x**2
    f = lambda x,y: x**2 + y**2
    print g(3)
    print g(1,2)
  

Input/Output

command line input/output

• print examples:

    # basics
    print "hello world"
    print "hello", "world"
    print "Power" + "God" # concatenating, no space in between
    word1 = "hello"; word2 = "world"
    print "say: %s %s" % (word1,word2)
    print "say: %s" % word1
    print word1, word2
  	
    # more examples
    print "God " *10 # repeatedly printing
    print """ 
    type in as long as you want
    you can use multiple lines
    """  # the use of tripple quotes
  

  • use %s for strings, %d for integers, %f for floating point. Use %r when debugging for raw version of variables
  • if add a , at the end of a print sentence, then it doesn’t end the line.

• taking inputs interactively:

    prompt = '>' # optional
    print "tell me your age: "
    age = raw_input(prompt)
    #eg
    age = raw_input("How old are you?")
    #or
    import sys
    data = sys.stdin.read() # read from input
  

  • you need to use print( ) in python3 or later versions

• passing command line input:

    from sys import argv
    # first argument is always the script
    script, first, second  = argv 	
    # first argument in arg1, the remaining in arg
    script, arg1, *arg = argv 
  

file input/output

• file objects:
  use txt = open(filename) where the open function returns a file object. There are also three modes, 'r'(read), 'w'(write), and 'a'(append), for opening the file. With the file object, you can do:

    txt.read()     
    txt.write("stuff")
    txt.readline() # read only one line   
    txt.truncate() # erase the file    
    txt.close() # close the file
  	
    # others
    txt.name  # name of the file
    txt.closed # bool: whether the file is closed
    txt.mode # open mode
  	
  

Control Flow

Indentation matters in the following codes.

# if sentence:
if cond1:
	statement1
elif cond2:
	statement2
else:
	statement3
	
# conditional expression	
x = val1 if condition else val2 # same as
if condition: x = val1
else: x = val2

	
# for loop:
for iterator in range:
	actions_in_each_iteration
else:
	after_condition_violation
	
# while loop:
while (condition):
	actions_in_each_iteration
else:
	after_condition_violation
	
# loop control statements
break
continue
pass # null operation

Note that an else: statement can be added after each looping structure.

Data Structures & Classes

Numbers

Tutorialspoint: Numbers overview

• type convertion functions:
  int(x); long(x); float(x); complex(x)

Strings

# initialization
mystring = "a test sentence"

# member functions
mystring.split(' ') # split by ' ' into a vector of strings
mystring.strip() # remove the return at the end (if any)
mystring.join(seq) # concatenate values in seq using mystring as separater
mystring.endswith(str) # test the end of mystring
mystring.replace(str1, str2) # replace str1 with str2, return new string
mystring.isupper(); mystring.islower() # test upper/lower case
mystring.index('a')
mystring.count('c') # get index and counts of a substring
mystring.find("str") # find substring; if not found return -1

# others
str[1:3] 
str[2:] # subsetting
str*2 # string repitition
str +  "another string"  # string concatenation
str() # convert to string
list(s) # turn string into list of characters

str[1]='a' #DOES NOT WORK! strings do not support modification
ord('x') # letter to ASCII code
chr(5) # code to letter

List

# initialization with []
mylist = [1,2,'a',(1,2,3),[1,2]]
mylist = ['a'] * 10  # initialize with length

# member functions
mylist.append() # append to the end of list
mylist.sort() # sort the list (in place)
mylist.pop()	# pop the last item
mylist.pop(i) # pop the ith
mylist.reverse()
mylist.insert(index, obj) # insertion 
mylist.remove(object) # deletion
mylist.count(item)  # count the frequency of item
mylist.index(val) # get index of a value
# ... etc

# others
mylist[2:]  # subsetting
mylist[-3:]
mylist[:2]
mylist[1:2]
mylist[::-1] # reversing
mylist*2 # repetition
mylist + list2 # list concatenation
del mylist[2] # item deletion
3 in [1,2,3]  # membership test
list(var) # convert to list

a =  b = c = []
# cautious! a,b,c refer to the same empty list

Note:

• list slicing results in a copy of list

Tuples

The main differences between lists and tuples are: Lists’ size can be changed, while tuples cannot be updated. Tuples can be thought of as read-only lists.

# initialization by round parenthesis
mytuple = ('abc','def',1,2,3)

# member functions
mytuple.index('abc')   # get index
mytuple.count(1)   # get frequency of element

# others (similar to lists)

Dictionary

Dictionary is a hash table like data structure in Python.

# initialization with curly braces
mydict  = {}
mydict = {'one': "power", 'two': "god"}

mydict['one'] = "item 1"
mydict[2] = "item 2" # keys can be almost anything that is immutable

# member functions
mydict.keys()  # get keys
mydict.values() # get values
mydict.get(key,default=None) # get value for key; if no key found, return default
mydict.clear()   # empty the mydict
mydict.items()   # returns a list of tuple (key, value) pairs
mydict.update(mydict2)  # add mydict2 elements to mydict; repeated keys in mydict2 will be discarded
mydict.pop(key) # remove an item
mydict.setdefault(key,default_value) # use default_value if key not found

Sets

# set initialization
s = set([...]) 
s = {0}

# operations
s.add('element') # add elements
s.remove("element")
s.union(s2)
s.intersection(s2)

# set operators
|, |=
&, &=
-, -=
^, ^=
>=, <=

CLASSES

Python is an object-oriented language, thus it’s very easy to define classes and objects.

## create a class
class myclass:
    "this is the first class I create"
    name = ""
    height = 0
    weight = 0
    bmi = 0
    __secret = "hidden variable"

    # class initiation method
    def __init__(self,name, height, weight):
        self.name = name
        self.height = height
        self.weight = weight
        self.bmi = (self.weight)/(self.height)**2

    # printing method
    def printmyclass(self):
        print("Name:",self.name)
		print("Height:",self.height,"\nWeight:",self.weight)
        print("Your bmi is calculated as: %.3f" % self.bmi)

	# destroy this object:
	# Python deletes built-in types or class instances automatically to free the memory space
	# just to pop out a message
	def __del__(self):
        print(self.__class__.__name__, "destroyed")
	
## use a class
me = myclass("PowerGod",1.72,68)
me.printmyclass() 
print me.bmi

## methods on a class object
print (hasattr(me,"height")) # test existence of an attribute
print (getattr(me,"height")) # get an attribute
setattr(me,"height",1.78)	# ...
delattr(me,"weight")
me.printmyclass()

## selected built-in attributes
me.__doc__   # documentation string
me.__dict__ # dictionary of class's namespace

# private variables
# name the variables you want to hide with double underscore prefix
print (me.__secret) # will give error

Class inheritance: assume that myclass is defined, then we can define a child subclass that inherit all its attributes:

class child(myclass):
    """
    this is a subclass of myclass
    all myclass's attributes can be inherited
    """
    major=""
    year=1993

    def printchild(self):
        print(self.name," is going to graduate in ",self.year)

me2 = child("PowerGod",1.00,45.00) # using parent class method
me2.printchild()  # using subclass method
me2.printmyclass() # using parent class method

# can also use super() to access parent class methods
# eg
	def __init__(self):
		super().__init__(a,b,c)

Notes:

• Python also allows multiple inheritance

  class child(myclass1, myclass2):
    pass
  

  child can use member methods from both parents. If there are shared method names, the method from myclass1 would be used

Python environment management

Assume you have Anaconda installed.

# create a new environment
conda create --name env_name python=3.5 ...
conda create --name a_copy python=2.7 anaconda # with all current packages, but on different version of python
conda create --name from_list --file requirements.txt

# activate and deactivate
source activate env_name
source deactivate

# list
conda search pandas
conda list pandas 
conda list --explicit > requirements.txt # dump packages in env

# install remove
conda install <package-name>
conda remove --name <env_name> --all # totally clear the environment

# info
conda info --env

Packages and Modules

Modules

# a python source file can be used as a module
# import a python source file support.py
import support  # similar to source(support.R)
support.myfun() # use myfun() from support.py

# import certain items from module
from module_name import item1, item2
from support import myfun # example
myfun()  # this time no need to use module name in the front

# import all names from the module
from moduel_name import * 
myfun()  # no need to use module name in the front

Packages
A package is a hierachical directory with multiple modules. For instance, I have a folder temp, with source files file1 and file2.

# to make it into a package, add a __init__.py file in the temp folder, with commands
__all__ = ["file1","file2"]
from temp.file1 import item1, item2 ...
from temp.file2 import item1, item2 ...

# now we can import this temp package 
import temp 
from temp import file1 # or part of the package
file1.item1()

# to skip temp.
from temp import *
file1.item1()
file2.item1()

• the __all__ is used for from temp import *. Without it, import * would result in name collision (item1 exists in both files). __all__ resolves this by requiring to add submodule name in the front.
• read more on organizing packages:
  python software fundation : modules

Useful Modules/Packages

• os: provide functions to take actions related to operating system

• sys: system-specific parameters and functions
  • sys.path ...: modify system searching path
  • sys.argv: command line inputs
  • sys.exit(code): exit program with code
• subprocess
  • subprocess.call(cmd, shell=True): run cmd as shell command
• re module for regular expressions:
  Python regular expression

Python for data analysis

The built-in data structures of Python are not easy to use when performing data analysis. A couple of popular packages provide Python many R-like functionalities, and are perfect for data scientists.

• NumPy for
  • math function evaluations (on arrays, list etc)
  • sampling from distribution (numpy.random.***)
  • array (matrix) manipulation
  • tutorial: NumPy quick start
• SciPy for scientific computing
  • linear algebra
  • numerical integration
  • optimization
• Pandas for
  • Series: fixed-size dict
  • DataFrame: most commonly used object
  • Pandas objects are suitable for most Numpy functions
  • import (or write) files from different data structures:
    • Series.to_csv, Series.from_csv
    • DataFrame.to_csv, DataFrame.from_csv
  • tutorial:
    pandas data structure
    pandas tutorial
• matplotlib for visualization
  • tutorial: matplotlib tutorial
• sklearn for
  • machine learning tools
• yaml for
  • read parameters from files:

    """
    The following is my parameter file:
    par1: a
    par2: [b,c]
    par3: (d,e,f)
    """
    import yaml
    txt = open(file_path)
    pars = yaml.load(txt)
    txt.close()
  

  Then the parameters are saved as elements in pars which is a dict type variable.

• pickle for saving and restoring python work space:

    import pickle
    # Save objects:
    f = open("results.pckl","wb")
    pickle.dump([obj1,obj2,obj3],f)
    f.close()
  	
    # restore the objects:
    f=open("results.pckl","rb")
    obj1,obj2,obj3 = pickle.load(f)
    f.close()
  

• data structure modules:
  • heapq for implementations of heaps

Selected topics

Sorting

# sort list using member function
# note: tuple, dict do not have .sort() member
ls =[a,b,c]
ls.sort() # default sorting
ls.sort(key = myfun) # customized sorting

# sort using sorted():
# note this can be used to sort tuple, dict and other objects
sorted(obj, key = myfun) 

Others

• get help:
  type pydoc python_function in terminal to get document for a python function

• IDE recommendation
  • Anaconda

• object ids (like address of an object in C):
  id(var), unique identification number for each object

• output formatting
  • link1 or link2
  • syntax for format placeholder {:[flags][width][.precision]type}
    • example: {:10.2f}.format(27.1205)
    • types: d, f, e, s
    • flags
      • left/right shift: <, >
      • center: ^
    • use variable: "{v:10d}".format(v=10)

• assertion:
  assert expression, "error message": if expression result in False, an error is incurred, and the error message is printed

• try-except exception handling, and raise exceptions

    # basic syntax
    try:
        code may have error
    except:
        what to do when error exists
  	
    # more detailed
    try:
       your operations ;
    except ExceptionI, ExceptionII:
       If there is ExceptionI, ExceptionI then execute this block.
    except ExceptionII:
       If there is ExceptionIII, then execute this block.
    else:
       If there is no exception then execute this block.
    finally:
        always execute this block
  
    # raise exceptions, eg:
    def raise_exception(s):
        if s<10:
            raise Errortype("error message")
        return
    try:
        raise_exception(20)
    except Errortype as e:
        actions on this exception(e)
        raise  
    ........
  	
    # define your own error type
    class my_err(Exception):
        def __init__(self,message):
            ...
        # usage
    try:
        raise my_err("message")
    except my_err as e:
        actions(e)
  

  for a list of Exception types, see exception handling

• reload module in interactive mode:
  when developing the package, you need to constantly modify and update the modules that have been imported. To re-import after modification, you need to use importlib.reload(module.file).

• Write python main function:
  Python does not have a int main() function as C/C++ do, but you can do the following to realize something similar to main():

    import XXXXX
    # you code here
    def myfun():
        # ....
    if __name__ == "__main__":
        # when file directly sourced
        # do something 
    else: 
        # when file imported by other processes
        # do soemthing else
  

  • variable __name__ represent the name of current module

• the with keyword

    # syntax
    with expression [as variable]:
   		with-block
   	# example 
   	with open('output.txt', 'w') as f:
   		f.write('Hi there!')
   	"""
   	1. can automatically close file at the end of with-block
   	2. guarantees to close file no matter what exception raised in the block
   	"""