cup.util package

Submodules

cup.util.conf module

author:Liu.Jia Guannan Ma create_date:2014 last_date:2014 descrition:Complex conf support

exception cup.util.conf.ArrayFormatError(errmsg)

Bases: cup.err.BaseCupException

数组类型错误

class cup.util.conf.CConf(path, name, revert_version='')

Bases: object

Depreciated class. Please do not use it. Use python configparser instead.

revert()

revert the conf

update(kvs)

update conf with a dict.

dict = {‘key’ : ‘value’, ‘key1’: ‘value’}

write_kv_into_conf(kvkvs)

将key-value写进conf

class cup.util.conf.CConfModer(toolpath)

Bases: object

历史遗留类. 请忽略. 操作我厂configure公共库conf的类。 主要调用cfmod这个工具进行. 推荐使用Configure2Dict转换成dict,在使用Dict2Configure类来操作conf文件. 如果只是简单的更新某个key, 可以使用此类.

addkv(confpath, key, val)

增加某个key/value到confpath

delkv(confpath, key)

删除confpath文件中的某个key

updatekv(confpath, key, val)

update key with value

updatekvlist(confpath, kvlist)

更新confpath文件里的内容。 kvlist是一个list. list里面每一个item是一个dict { ‘key’ : ‘xxx:xx:xx’, ‘value’ : ‘updated’}

class cup.util.conf.ConfDict(*args, **kwargs)

Bases: dict

ConfDict that Configure2Dict and Dict2Configure can use.

get_ex(key)

get (value, comments) with key, comments is a list

get_ordered_keys()

get keys in order

set_ex(key, value, comments)

In addtion to dict[‘key’] = value, set_ex also set comments along with the key.

exception cup.util.conf.ConfDictSetItemError(errmsg)

Bases: cup.err.BaseCupException

ConfDict Error

class cup.util.conf.ConfList

Bases: list

增加一个conf的数组属性. 以ConfList的数据方式表现

e.g.

@disk: /home/disk1 @disk: /home/disk2

append(item)

append_ex(item, comments)

append a item with conf comments

extend(seqs)

get_ex(ind)

get conf list item with its comments

insert(ind, item)

exception cup.util.conf.ConfListSetItemError(errmsg)

Bases: cup.err.BaseCupException

ConfList Error

class cup.util.conf.Configure2Dict(configure_file, remove_comments=True)

Bases: object

Configure2Dict support conf features below:

1. comments

   As we support access/modify comments in a conf file, you should obey rules below:

   Comment closely above the object you want to comment. Do NOT comment after the line.

   Otherwise, you might get/set a wrong comment above the object.

2. sections

2.1 global section

• if key:value is not under any [section], it is under the global layer

  by default

• global section is the 0th layer section

e.g. test.conf:

# test.conf global-key: value global-key1: value1

2.2 child section

• [section1] means a child section under Global. And it’s the

  1st layer section

• [.section2] means a child section under the nearest section

  above. It’s the 2nd layer section.

• [..section3] means a child section under the nearest section

  above. And the prefix .. means it is the 3rd layer section

e.g.: test.conf:

global-key: value [section]

host: abc.com port: 8080 [.section_child]

child_key: child_value [..section_child_child]

control: ssh [...section_child_child_child]

wow_key: wow_value

2.3 section access method

get_dict method will convert conf into a ConfDict which is derived

from python dict.

• Access the section with confdict[‘section’][‘section-child’].

• Access the section with confdict.get_ex(‘section’) with (value,

  comments)

3. key:value and key:value array

3.1 key:value

key:value can be set under Global section which is closely after the 1st line with no [section] above.

key:value can also be set under sections.

E.g.

# test.conf key1: value1 [section]

key_section: value_in_section [.seciton]

key_section_child: value_section_child

3.2 key:value arrays

key:value arrays can be access with confdict[‘section’][‘disk’].

You will get a ConfList derived from python list.

# test.conf # Global layer, key:value host: abc.com port: 12345 # 1st layer [monitor] @disk: /home/data0 @disk: /home/data1 [section]

@disk: /home/disk/disk1 @disk: /home/disk/disk2

4. Example

   # test.conf # Global layer, key:value host: abc.com port: 12345 # 1st layer [monitor] @disk: /home/data0 @disk: /home/data1 [section]

   @disk: /home/disk/disk1 @disk: /home/disk/disk2

   [monitor]

   timeout: 100 regex: sshd # 2nd layer that belongs to [monitor] [.timeout]

   # key:value in timeout max: 100 # 3rd layer that belongs to [monitor] [timeout] [..handler]

   default: exit

get_dict()

get conf_dict which you can use to access conf info

class cup.util.conf.Dict2Configure(conf_dict)

Bases: object

Convert Dict into Configure. You can convert a ConfDict or python dict into a conf file.

set_dict(conf_dict)

set a new conf_dict

write_conf(conf_file)

write the conf into of the dict into a conf_file

exception cup.util.conf.KeyFormatError(errmsg)

Bases: cup.err.BaseCupException

Key error class

exception cup.util.conf.LineFormatError(errmsg)

Bases: cup.err.BaseCupException

Line error class

exception cup.util.conf.UnknowConfError(errmsg)

Bases: cup.err.BaseCupException

unkown error class

exception cup.util.conf.ValueFormatError(errmsg)

Bases: cup.err.BaseCupException

value error class

cup.util.constants module

descrition:heartbeat related module test-case-name: twisted.python.test.test_constants Copyright (c) Twisted Matrix Laboratories. See LICENSE for details. license:CUP dev team modified and redistrbuted this module, according to MIT license(http://opensource.org/licenses/mit-license.php) that Twisted lib obeys. If any concern, plz contact mythmgn@gmail.com.

class cup.util.constants.NamedConstant

Bases: cup.util.constants._Constant

L{NamedConstant} defines an attribute to be a named constant within a collection defined by a L{Names} subclass.

L{NamedConstant} is only for use in the definition of L{Names} subclasses. Do not instantiate L{NamedConstant} elsewhere and do not subclass it.

class cup.util.constants.ValueConstant(value)

Bases: cup.util.constants._Constant

L{ValueConstant} defines an attribute to be a named constant within a collection defined by a L{Values} subclass.

L{ValueConstant} is only for use in the definition of L{Values} subclasses. Do not instantiate L{ValueConstant} elsewhere and do not subclass it.

class cup.util.constants.FlagConstant(value=None)

Bases: cup.util.constants._Constant

L{FlagConstant} defines an attribute to be a flag constant within a collection defined by a L{Flags} subclass.

L{FlagConstant} is only for use in the definition of L{Flags} subclasses. Do not instantiate L{FlagConstant} elsewhere and do not subclass it.

class cup.util.constants.Names

Bases: cup.util.constants._ConstantsContainer

A L{Names} subclass contains constants which differ only in their names and identities.

class cup.util.constants.Values

Bases: cup.util.constants._ConstantsContainer

A L{Values} subclass contains constants which are associated with arbitrary values.

classmethod lookupByValue(value)

Retrieve a constant by its value or raise a C{ValueError} if there is no constant associated with that value.

@param value: The value of one of the constants defined by C{cls}.

@raise ValueError: If C{value} is not the value of one of the constants

defined by C{cls}.

@return: The L{ValueConstant} associated with C{value}.

class cup.util.constants.Flags

Bases: cup.util.constants.Values

A L{Flags} subclass contains constants which can be combined using the common bitwise operators (C{|}, C{&}, etc) similar to a I{bitvector} from a language like C.

cup.util.context module

author:Guannan Ma create_date:2014 last_date:2014 descrition:context for threadpool

class cup.util.context.ContextManager

Bases: object

函数调用上下文相关类。

call_with_context(new_context, func, *args, **kwargs)

context is a {}

get_context(key)

get the context that has the key

class cup.util.context.ContextTracker4Thread

Bases: object

进行线程的上下文切换相关class.

call_with_context(context, func, *args, **kwargs)

调用函数func, 并使用当前context

current_context()

get current context

get_context(key)

获得某个key的对应的context

cup.util.generator module

author:Guannan Ma create_date:2014 last_date:2014

class CGeneratorMan(object)

用来生成各类唯一数，字符集，线程安全的自增uint的类。 目前生成函数较少， 欢迎大家贡献ci. Singleton类。初始化需要传入一个用来生成字符集的string.

初始化函数:

__init__(self, str_prefix=get_local_hostname())

默认传入当前自己的hostname

成员函数:

get_uniqname()

获得一个以传入的初始化string为base, 加上pid, threadid等组合的host级别的 unique string.

get_next_uniq_num()

获得当前进程唯一的自增非负整数。线程安全.

cup.util.misc module

author:Guannan Ma create_date:2014 last_date:2014 descrition:misc functions

class cup.util.misc.CAck(binit=False)

Bases: object

Ack, 可设置bool值并获取

getack_infobool()

取得ack的bool返回值

setack_infobool(binit=False)

设置ack的bool值

cup.util.misc.check_not_none(param)

检查param不是None, 如果是None, raise NameError

cup.util.misc.check_type(param, expect)

检查param是否和expect一样的类型。如果不一样raise TypeError

cup.util.misc.get_filename(backstep=0)

获得调用该函数的代码行所在的文件名。 backstep代表是否将调用栈增/减.

cup.util.misc.get_funcname(backstep=0)

获得调用该函数的代码行所在的函数名。 backstep代表是否将调用栈增/减. backstep默认0

cup.util.misc.get_lineno(backstep=0)

获得调用该函数的代码行.

cup.util.thread module

author:Zhaominghao Guannan Ma create_date:2014 last_date:2014 descrition:cup thread module

cup.util.thread.async_raise(tid, exctype)

Raises an exception in the threads with id tid

class cup.util.thread.CupThread(group=None, target=None, name=None, args=(), kwargs=None, verbose=None)

Bases: threading.Thread

CupThread继承threading.Thread, 支持threading.Thread所有功能和特性, CupThread扩展了三个功能,raise_exc给线程发送raise信号,get_my_tid返回线程id， terminate同步中止线程

get_my_tid()

返回线程id

raise_exc(exctype)

异步给线程发送raise，发送成功返回1，线程已经停止返回0，其他错误返回!=1 :param exctype:

raise Exception, exctype type is class

terminate(times=15)

异步raise线程，尝试中止线程 中止成功返回True， 线程已经停止返回True， 停止失败返回False。 (在返回True的时候，表示线程已经被中止, 在中止失败的过程中，会重试times次)

class cup.util.thread.RWLock

Bases: object

读写锁类

acquire_readlock(wait_time=None)

获取读锁， 如果wait_time赋值且!=None的数，会等待wait_time. 如果之后还没拿到锁， 将raise RuntimeError

acquire_writelock(wait_time=None)

获取写锁， 如果wait_time赋值且!=None的数，会等待wait_time. 如果之后还没拿到锁， 将raise RuntimeError

release_readlock()

释放读锁

release_writelock()

释放写锁

cup.util.threadpool module

author:Guannan Ma create_date:2014 last_date:2014 descrition:Guannan ported threadpool from twisted.python. Mit License applied for twisted. http://www.opensource.org/licenses/mit-license.php if any concern, plz contact mythmgn@gmail.com

class cup.util.threadpool.ThreadPool(minthreads=5, maxthreads=20, name=None)

Bases: object

Threadpool class

add_1job(func, *args, **kwargs)

Parameters:

• func – 会被线程池调度的函数
• *args –

  func函数需要的参数

• **kw –

  func函数需要的kwargs参数

add_1job_with_callback(result_callback, func, *args, **kwargs)

Parameters:

• result_callback –

  func作业处理函数被线程池调用后，无论成功与否都会 执行result_callback.

  result_callback函数需要有两个参数 (ret_in_bool, result), 成功的话为(True, result), 失败的话 为(False, result)

  如果func raise exception, result_callback会收到(False, failure)

• func – 同add_1job, 被调度的作业函数
• *args –

  同add_1job, func的参数

• **kwargs –

  同add_1job, func的kwargs参数

adjust_poolsize(minthreads=None, maxthreads=None)

调整线程池的线程最少和最多运行线程个数

dump_stats()

打印当前threadpool的状态信息到log 和stdout 其中状态信息来自于get_stats函数

get_stats()

回返当前threadpool的状态信息. 其中queue_len为当前threadpool排队的作业长度 waiters_num为当前空闲的thread num working_num为当前正在工作的thread num thread_num为当前一共可以使用的thread num:

stat = {}
stat['queue_len'] = self._jobqueue.qsize()
stat['waiters_num'] = len(self._waiters)
stat['working_num'] = len(self._working)
stat['thread_num'] = len(self._threads)

start()

启动线程池

start1worker()

为线程池增加一个线程。

stop()

停止线程池， 该操作是同步操作， 会夯住一直等到线程池所有线程退出。

stop1worker()

为线程池减少一个线程。

try_stop(check_interval=0.1)

发送停止线程池命令， 并尝试查看是否stop了。 如果没停止，返回False

try_stop不会夯住， 会回返。 属于nonblocking模式下

Module contents

author:Guannan Ma maguannan@baidu.com @mythmgn create_date:2014 last_date:2014 descrition:utilization related module

cup.util.check_not_none(param)

同cup.util.misc.check_type. 请使用前者

cup.util.check_type(param, expect)

同cup.util.misc.check_type. 请使用前者