Matthew Note

Python Notes

|
Python 相关笔记

Pypi镜像

解决国内访问pypi速度慢的问题，注意要加上--trusted-host pypi.mirrors.ustc.edu.cn 因为有些不提供安全连接，或者证书不可信

1
2
3
4
http://pypi.douban.com/simple  豆瓣
http://pypi.hustunique.com/simple  华中理工大学
http://pypi.sdutlinux.org/simple 山东理工大学
http://pypi.mirrors.ustc.edu.cn/simple 中国科学技术大学
1
pip3.4 install -i http://pypi.mirrors.ustc.edu.cn/simple --trusted-host pypi.mirrors.ustc.edu.cn  flask-restful

模块

from import

  • 如果module1中有变量X，module2用from来导入module1，则在赋值X的时候并不会改动module1中的X，而用import可以。
  • from会导致reload失效，因为from导入的模块拿的只是一个被reload的引用，必须在reload之后再执行一次from import

类

  • 子类并不会自动调用父类的构造函数，需要显示的调用。
  • 类的赋值操作并不会查询搜索树。
  • init 函数不支持重载，因为python支持可变参数
  • 两个下划线开头的是伪私有类属性, 他会变成_ClassName__X

内建属性

1
2
3
4
5
6
7
8
__class__ # 返回类名
__base__ # 父类
__dict__ # 属性字典，dir()
__slot__ # 禁止生成dict树
__getattr__ # X.undefined
__getattribute__ # X.any
__contain__ # item in X
__call__ # 可以直接这样使用 class_object_name()

委托

1
2
3
4
5
6
7
class Super:
    def delegate(self):
        self.action() #action需要被实现
        
class Provider(Super):
    def action(self):
        print('in provider.action')

函数

闭包

1
2
3
4
5
def func():
    _x = "attr"
    def printf():
        print(_x)
    return printf

Decorator

Singleton

1
2
3
4
5
6
7
8
9
10
def singleton(cls, *args, **kw):   
    _instance = {} 
    def wrapper(): 
        if cls not in _instance:   
            _instance[cls] = cls(*args, **kw)   
        return _instance[cls]      
    return wrapper                 
@singleton 
class MyClass(gentoo.LXRPClientHandler):
  • 缺点是MyClass实际上是一个函数（wrapper），所以不能直接通过MyClass取类中属性

  • 且m = MyClass(); n = MyClass(); o = type(n)(); then m == n && m != o && n != o

  • @staticmethod不需要表示自身对象的self和自身类的cls参数，就跟使用函数一样。

  • @classmethod也不需要self参数，但第一个参数需要是表示自身类的cls参数。

规范

命名规则

Type Public Internal
Modules lower_with_under _lower_with_under
Packages lower_with_under
Classes CapWords _CapWords
Exceptions CapWords
Functions lower_with_under() _lower_with_under()
Global/Class Constants CAPS_WITH_UNDER _CAPS_WITH_UNDER
Global/Class Variables lower_with_under _lower_with_under
Instance Variables lower_with_under _lower_with_under (protected) or __lower_with_under (private)
Method Names lower_with_under() _lower_with_under() (protected) or __lower_with_under() (private)
Function/Method Parameters lower_with_under
Local Variables lower_with_under
  1. single_trailing_underscore_ : used by convention to avoid conflicts with Python keyword, e.g.
1
Tkinter.Toplevel(master, class_='ClassName')

  1. __double_leading_underscore : when naming a class attribute, invokes name mangling (inside class FooBar, boo becomes _FooBarboo).

  2. __double_leading_and_trailing_underscore__ : “magic” objects or attributes that live in user-controlled namespaces. E.g. init , import or file . Never invent such names; only use them as documented.

  3. _single_leading_underscore : weak “internal use” indicator. E.g. from M import * does not import objects whose name starts with an underscore.

编程规范

  • 每行80字符，注释72字符
  • 参数多的时候要换行，少数情况下要用\
  • 使用 if not alist 检查是否为空，而不是用if len(alist) == 0

import顺序

不要一行import多个包

Imports should be grouped in the following order:

  1. standard library imports
  2. related third party imports
  3. local application/library specific imports
  4. Wildcard imports ( from import * ) should be avoided

类设计

如果在不确定函数或者属性是public还是non-public，尽量还是先声明为non-public，从non到public很容易，反过来则很难

Programming Recommendations

1
2
3
4
5
6
7
Yes:
def f(x): return 2*x
No:
f = lambda x: 2*x

The first form means that the name of the resulting function object is specifically ‘f’ instead of the generic ‘‘. This is more useful for tracebacks and string representations in general. The use of the assignment statement eliminates the sole benefit a lambda expression can offer over an explicit def statement (i.e. that it can be embedded inside a larger expression)

  • Use exception chaining appropriately. In Python 3, “raise X from Y” should be used to indicate explicit replacement without losing the original traceback.
  • Derive exceptions from Exception rather than BaseException . Direct inheritance from BaseException is reserved for exceptions where catching them is almost always the wrong thing to do.
  • Additionally, for all try/except clauses, limit the try clause to the absolute minimum amount of code necessary. Again, this avoids masking bugs.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
Yes:
try:
    value = collection[key]
except KeyError:
    return key_not_found(key)
else:
    return handle_value(value)
No:
try:
    # Too broad!
    return handle_value(collection[key])
except KeyError:
    # Will also catch KeyError raised by handle_value()
    return key_not_found(key)
  • Object type comparisons should always use isinstance() instead of comparing types directly.
1
2
3
Yes: if isinstance(obj, int):
No:  if type(obj) is type(1):
  • For sequences, (strings, lists, tuples), use the fact that empty sequences are false.
1
2
3
4
5
Yes: if not seq:
     if seq:
No: if len(seq)
    if not len(seq)
  • Don’t compare boolean values to True or False using == .
1
2
3
Yes:   if greeting:
No:    if greeting == True:
Worse: if greeting is True:

Class 继承

当一个类被设计为基类的时候，要考虑到那些方法是要被子类覆盖的，那些是给子类用的（subclass API），那些只是基类用的

  • 公共属性，前面不应该有一个下划线
  • 如果你的属性名与keyword或者其他什么的冲突，则可以考虑在尾部加一个下划线
  • 对于简单的公共属性，直接用属性名访问，而不要像jave一样定义一些accessor/mutator methods，如果之后需要对于这个属性进行额外处理， 那么可以使用@property来强化这个属性（防止用property来计算复杂属性）
  • 如果你的类可能被继承，则把那些不想被子类使用的属性和方法用两个下划线开头

super() 还是 BaseClass()

1
2
3
4
5
6
7
8
class Base:
    name = "Base"
    def __init__(self):pass
    
class Sub(Base):
    def __init__(self):
        #Base.__init__(self)
        super().__init__()

这两种方法都可以用作调用父类初始化函数，但是super()更好，不会在菱形继承中导致一些重复初始化的问题

为了弄清它的原理，我们需要花点时间解释下Python是如何实现继承的。对于你定义的每一个类而已，Python会计算出一个所谓的方法解析顺序(MRO)列表。 这个MRO列表就是一个简单的所有基类的线性顺序表。例如：

1
2
3
C.__mro__
(<class '__main__.C'>, <class '__main__.A'>, <class '__main__.B'>,
<class '__main__.Base'>, <class 'object'>)

为了实现继承，Python会在MRO列表上从左到右开始查找基类，直到找到第一个匹配这个属性的类为止。

而这个MRO列表的构造是通过一个C3线性化算法来实现的。 我们不去深究这个算法的数学原理，它实际上就是合并所有父类的MRO列表并遵循如下三条准则：

  • 子类会先于父类被检查
  • 多个父类会根据它们在列表中的顺序被检查
  • 如果对下一个类存在两个合法的选择，选择第一个父类

当你使用 super() 函数时，Python会在MRO列表上继续搜索下一个类。 只要每个重定义的方法统一使用 super() 并只调用它一次， 那么控制流最终会遍历完整个MRO列表，每个方法也只会被调用一次。

我们来看下super的实现，可以发现他在找这个列表里后一个class

1
2
3
def super(cls, inst):
    mro = inst.__class__mro()
    return mro[mro.index(cls)+1]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
class Base:
    name = "Base"
    def __init__(self):pass
a = Base()
print(a.name)
a.name = "Base c"
print(a.name)
print(Base.name)
print('--------')
class Sub(Base):
    def __init__(self):
        Base.__init__(self)
    
    @property
    def name(self):
        print('property name')
        print(Base.name)
        Base.name = 'abc'
        print(super(Sub,self).name)
b = Sub()
b.name
print(b.__dict__)
print(Base.name)
print('--------')
class Sub2(Sub):
    def __init__(self):
        Base.__init__(self)
        print(super(Sub, self).name)
c = Sub2()
c.name
print(c.__class__.mro())

结果是：

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
Base
Base c
Base
--------
property name
Base
abc
{}
abc
--------
abc
property name
abc
abc
[<class '__main__.Sub2'>, <class '__main__.Sub'>, <class '__main__.Base'>, <class 'object'>]

weakref

List和Dict不支持weakref，但是他们的子类可以

Unicode

为了解决多国语言的问题，ASCII显然已经不适合，他是用一个字节代表一个字符，这远远不够世界各种语言使用，Unicode是一个更大的集合，他包含了ASCII，因此他是一个变长的编码方式
我们有两种方式来声明一个str

1
2
str = ‘abcd’ # 这是一个str
uni = u‘Hi \u2119\u2602’ # 这是一个unicode类型
  • unicode.encode(‘UTF-8’) -> bytes
  • bytes.decode(‘UTF-8’) -> unicode
1
2
3
u'abcd'.encode('ascii', 'replace') #不能被encode就会变成一个`?`
u'abcd'.encode('ascii', 'xmlcharrefreplace') #produces an HTML/XML character entity reference
u'abcd'.encode('ascii', 'ignore') #ignore
  • Python 2 will automatically decode the byte string to produce a second unicode string, then will complete the operation with the two unicode strings.
  • So “str” in Python 2 is now called “bytes,” and “unicode” in Python 2 is now called “str”. This makes more sense than the Python 2 names, since Unicode is how you want all text stored, and byte strings are only for when you are dealing with bytes.
    [Pragmatic Unicode][http://nedbatchelder.com/text/unipain.html]

元类

type是所有类的元类(metaclass)

TIP

in

if elem in alist 用来检查元素是否在列表里，但是注意这里的elem必须是一个元素，不能是alist列表中的一个子集，他是不能够检查子集的。

yeild

generator可以包含一个return语句，但是他会导致generator不再能返回任何结果，而且return不能有返回值，return 5会报一个syntax error

_ 占位符

可以像Haskell一样有个占位符_ 来表示不关心的内容

Flask

获得message body

1
2
3
request.form
request.json
request.headers

Functional programming

generator expression and list comprehensions

1
2
3
4
5
# Generator expression -- returns iterator
stripped_iter = (line.strip() for line in line_list)
# List comprehension -- returns list
stripped_list = [line.strip() for line in line_list]

List comprehension 会一次性返回所有的值，而generator expression返回一个可迭代的对象，他不会一次性返回所有值，只会根据需要获取，适合处理无限长度的数据。

Passing values into a generator

In Python 2.4 and earlier, generators only produced output. Once a generator’s code was invoked to create an iterator, there was no way to pass any new information into the function when its execution is resumed. You could hack together this ability by making the generator look at a global variable or by passing in some mutable object that callers then modify, but these approaches are messy.
In Python 2.5 there’s a simple way to pass values into a generator. yield became an expression, returning a value that can be assigned to a variable or otherwise operated on:

1
val = (yield i)

I recommend that you always put parentheses around a yield expression when you’re doing something with the returned value, as in the above example. The parentheses aren’t always necessary, but it’s easier to always add them instead of having to remember when they’re needed.

(PEP 342 explains the exact rules, which are that a yield-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment. This means you can write val = yield i but have to use parentheses when there’s an operation, as in val = (yield i) + 12.)

Values are sent into a generator by calling its send(value) method. This method resumes the generator’s code and the yield expression returns the specified value. If the regular next() method is called, the yield returns None.

Here’s a simple counter that increments by 1 and allows changing the value of the internal counter.

1
2
3
4
5
6
7
8
9
def counter (maximum):
    i = 0
    while i < maximum:
        val = (yield i)
        # If value provided, change counter
        if val is not None:
            i = val
        else:
            i += 1

And here’s an example of changing the counter:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
>>>
>>> it = counter(10)
>>> print it.next()
0
>>> print it.next()
1
>>> print it.send(8)
8
>>> print it.next()
9
>>> print it.next()
Traceback (most recent call last):
  File "t.py", line 15, in ?
    print it.next()
StopIteration

Because yield will often be returning None, you should always check for this case. Don’t just use its value in expressions unless you’re sure that the send() method will be the only method used to resume your generator function.

In addition to send(), there are two other new methods on generators:

  • throw(type, value=None, traceback=None) is used to raise an exception inside the generator; the exception is raised by the yield expression where the generator’s execution is paused.

  • close() raises a GeneratorExit exception inside the generator to terminate the iteration. On receiving this exception, the generator’s code must either raise GeneratorExit or StopIteration; catching the exception and doing anything else is illegal and will trigger a RuntimeError. close() will also be called by Python’s garbage collector when the generator is garbage-collected.

If you need to run cleanup code when a GeneratorExit occurs, I suggest using a try: … finally: suite instead of catching GeneratorExit.

The cumulative effect of these changes is to turn generators from one-way producers of information into both producers and consumers.

Generators also become coroutines, a more generalized form of subroutines. Subroutines are entered at one point and exited at another point (the top of the function, and a return statement), but coroutines can be entered, exited, and resumed at many different points (the yield statements).

getattr

  • getattr(object, name[, default])
    Return the value of the named attribute of object. name must be a string. If the string is the name of one of the object’s attributes, the result is the value of that attribute. For example, getattr(x, 'foobar') is equivalent to x.foobar. If the named attribute does not exist, default is returned if provided, otherwise AttributeError is raised.

class type(object)

  • class type(object)
  • class type(name, bases, dict)
    With one argument, return the type of an object. The return value is a type object and generally the same object as returned by object.class.

The isinstance() built-in function is recommended for testing the type of an object, because it takes subclasses into account.

With three arguments, return a new type object. This is essentially a dynamic form of the class statement. The name string is the class name and becomes the name attribute; the bases tuple itemizes the base classes and becomes the bases attribute; and the dict dictionary is the namespace containing definitions for class body and becomes the dict attribute. For example, the following two statements create identical type objects:

1
2
3
4
5
>>>
>>> class X:
...     a = 1
...
>>> X = type('X', (object,), dict(a=1))

monkey patch

monkey patch指的是在运行时动态替换,一般是在startup的时候.

用过gevent就会知道,会在最开头的地方gevent.monkey.patch_all();把标准库中的thread/socket等给替换掉.这样我们在后面使用socket的时候可以跟平常一样使用,无需修改任何代码,但是它变成非阻塞的了. 之前做的一个游戏服务器,很多地方用的import json,后来发现ujson比自带json快了N倍,于是问题来了,难道几十个文件要一个个把import json改成import ujson as json吗? 其实只需要在进程startup的地方monkey patch就行了.是影响整个进程空间的.

同一进程空间中一个module只会被运行一次.

1
2
3
4
5
6
7
8
9
10
import json
import ujson
def monkey_patch_json():
  json.__name__ = 'ujson'
  json.dumps = ujson.dumps
  json.loads = ujson.loads
monkey_patch_json()
print 'main.py',json.__name__
import sub

sub.py

1
2
import json
print 'sub.py',json.__name__

with

如下情况 with一旦结束，那么as后面的对象就会被清除，即便是赋值给了c.response，那只是一个引用，除非用deepcopy

1
2
3
4
5
6
with urllib.request.urlopen(c.request, context=context) as response:
    c.response = response
    return c
#print(c.response.read())
#return c

pdb

可以通过代用set_trace()来设置一个断点，然后运行的时候就会自动进入pdb，然后就可以调试啦

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
import inspect
import pdb
class a:pass
class b(a):
    class c(a):
        def hell(self):
            print(inspect.getmodule(self))
        def guess(self):
            self.hell()
    def __call__(self):
        cc = self.c()
        pdb.set_trace()
        cc.guess()
cc()

函数式编程

map

从python3开始map变得lazy，如果你不主动调用next或者使用它 它是不会马上返回结果的，跟Haskell一样

1
2
3
4
5
6
7
8
class Test：
	def __init__(self, string)
		print(string)
		
x = map(Test, ['ggg','rrr']) //return only a map object
x.next()
list(x)// will get a result.

TIP

OrderedDict.move_to_end(key, last=False)

这种方式并不能用于排序，在把某个元素放到最后的同时也会改变其他元素的顺序

1
2
3
4
x=OrderedDict({'a':1,'b':2,'c':3})
x.move_to_end('a')
print(x)
>>OrderedDict([('c', 3), ('b', 2), ('a', 1)])

raise new_exc from original_exc

When raising a new exception (rather than using a bare raise to re-raise the exception currently being handled), the implicit exception context can be supplemented with an explicit cause by using from with raise:

Reference

  1. PEP 0008 – Style Guide for Python Code