utils Package

utils Package

debug Module

Utilities useful in debugging

pyfusion.utils.debug.equal_except_for(a, b, except_arg=None)

Compare objects via __dict__. allow for ignored attributes.

(Ugly code - but it works)

dist_mp Module

Multiprocessing distance calculations for clustering Return for multithreading based on last example in http://stackoverflow.com/questions/6893968/how-to-get-the-return-value-from-a-thread-in-python

from test_dist.py

class pyfusion.utils.dist_mp.ThreadWorker(func)

The basic idea is given a function create an object. The object can then run the function in a thread. It provides a wrapper to start it,check its status,and get data out the function.

get_results()

save_data(func)

modify function to save its returned data

start(params, **kwargs)

status()

pyfusion.utils.dist_mp.dist_mp(cl_instance, instances, squared=None, averaged=None, threads=None, debug=0)

multithreaded distance calculation using dist.pyx and nogil version set debug>=1 to to a comparison against single processor, GIL version

>>> import numpy as np
>>> x=dist_mp(np.random.random((14,)).astype(np.float32),np.random.random((1000,14)).astype(np.float32),debug=1,threads=3)

fftw3_bdb_utils Module

pyfusion.utils.fftw3_bdb_utils.load_wisdom(filenames=['/home/bdb112/.pyfftw/wisdom_d', '/home/bdb112/.pyfftw/wisdom_f', '/home/bdb112/.pyfftw/wisdom_l'])

pyfusion.utils.fftw3_bdb_utils.save_wisdom(filenames=['/home/bdb112/.pyfftw/wisdom_d', '/home/bdb112/.pyfftw/wisdom_f', '/home/bdb112/.pyfftw/wisdom_l'])

Save the wisdom accumulated since the last load_wisdom, which occurs when pyfusion is imported Best to run typical codes before saving. To see the current state of wisdom !fftw-wisdom - or look at the file. I guess these are just timings for the routines tested e.g extensive tests of a 32 element complex forward out of place transform.

!fftw-wisdom -x cof32

(fftw-3.3.3 fftw_wisdom #x458a31c8 #x92381c4c #x4f974889 #xcd46f97e

(fftw_codelet_t2fv_4_avx 0 #x1040 #x1040 #x0 #x12dc3d8e #xe40293c9 #x508e7f21 #x18911bc9) (fftw_codelet_n2fv_8_avx 0 #x1040 #x1040 #x0 #xb916fb98 #xf394dae5 #xe9a593f6 #x4ce2ac3f)

) for in place, just (fftw-3.3.3 fftw_wisdom #x458a31c8 #x92381c4c #x4f974889 #xcd46f97e

(fftw_codelet_n2fv_32_sse2 0 #x1040 #x1040 #x0 #x8ee86d9a #x3bf651fe #x73d5cbe4 #xfd6f3826)

)

host Module

pyfusion.utils.host.host()

net Module

Network utilities

pyfusion.utils.net.ping()

primefactors Module

find prime factors of a number, especially to allow estimation of the time taken by an FFT

For fast methods up to ~ 10^6 (e.g. numpy http://stackoverflow.com/questions/2068372/fastest-way-to-list-all-primes-below-n-in-python

For methods with good scaling: http://stackoverflow.com/questions/2211990/how-to-implement-an-efficient-infinite-generator-of-prime-numbers-in-python/

primesfrom2to3 (numpy is fastest for 1e9: 9secs, and rwh_primes2 47s pure see http://www.fftw.org/speed MFlops ~ 5*N*log2(N)/fft_time_estimate(N)/1e6 for complex fft ~ 1399 for 32768 fftw3 is ~ 2x faster than fftw2 (more for large arrays, less for small (<100k elts)

These initial attempts were not efficient - pyfftw is much better speedup is 3-10x /bin/fftw3bench cf$((32768+37)) 1.08ms on E4300 fftw3, 1.66 for numpy improve a little with “exhaustive” on lenny, but no improvement on laptop ./bench -oexhaustive cif$((32768+37)) Problem: cif32805, setup: 34.59 s, time: 974.75 us, ‘’mflops’‘: 2524.4

larger FFT lenny: cif$((32768*256)) 519.58 ms, ‘’mflops’‘: 1856.7 (8M samples) E4300 724.98 ms, ‘’mflops’‘: 1330.6

IDL 8 fft is similar speed, but not for some odd sizes (e.g. 2**15+37)

Speeds with ./configure –enable-sse2 bdb112@lenny:/tmp/fftw3/fftw-3.3.3/tests$ ./bench cif$((32768)) Problem: cif32768, setup: 1.79 s, time: 569.31 us, ‘’mflops’‘: 4316.8 bdb112@lenny:/tmp/fftw3/fftw-3.3.3/tests$ ./bench -onosimd cif$((32768)) Problem: cif32768, setup: 1.12 s, time: 816.06 us, ‘’mflops’‘: 3011.5 bdb112@lenny:/tmp/fftw3/fftw-3.3.3/tests$ ./bench -onosimd cif$((32768*32)) Problem: cif1048576, setup: 2.34 s, time: 50.32 ms, ‘’mflops’‘: 2083.9 bdb112@lenny:/tmp/fftw3/fftw-3.3.3/tests$ ./bench cif$((32768*32)) Problem: cif1048576, setup: 2.00 s, time: 39.95 ms, ‘’mflops’‘: 2625

./configure –enable-sse2 –with-combined-threads –enable-threads 2 threads -> 1.65x - 3 a tiny bit faster, 4 same as 2threads Problem: cif1048576, setup: 1.93 s, time: 40.18 ms, ‘’mflops’‘: 2609.6 bdb112@lenny:/tmp/fftw3/fftw-3.3.3$ tests/bench -onthreads=2 cif$((32768*32)) Problem: cif1048576, setup: 1.68 s, time: 24.78 ms, ‘’mflops’‘: 4230.7

8M elts 2 is 1.8x faster, 4 no better. bdb112@lenny:/tmp/fftw3/fftw-3.3.3$ tests/bench -onthreads=1 cif$((32768*256)) Problem: cif8388608, setup: 96.84 s, time: 361.73 ms, ‘’mflops’‘: 2666.9 estimate only gives 486 ms bdb112@lenny:/tmp/fftw3/fftw-3.3.3$ tests/bench -onthreads=2 cif$((32768*256)) Problem: cif8388608, setup: 72.35 s, time: 204.83 ms, ‘’mflops’‘: 4709.6

-nosimd c128 (r64,i64) 50ms for cif1048576 cf 37ms. c64 is 1.9x faste -> 27ms

pyfusion.utils.primefactors.fft_time_estimate(n, fft_type='numpy', plimit=30)

time in sec fft_type = ‘numpy’,’fftw3’ or ‘scipy.hilbert’

plimit prevents wasting time on large prime factors Saving is typically only a factor of up to 2 or 3 because the only difference is when the second largest PF is gt plimit and there are few of these, so plimit=30 is fine. For example the tests below (near 32768) work fine with 30

pyfusion.utils.primefactors.get_fftw3_speed(arr, iters=10, direction=None, dtype=<type 'numpy.float32'>, **kwargs)

measure the fftw3 speed for various data sizes by using plan with estimate, and running one instance. If arr is int, then the elements are different array sizes, otherwise use the array. direction default - just fwd - use ‘both’ for both To “train”: - the print allows you to ^c out. from pyfusion.utils.primefactors import get_fftw3_speed from pyfusion.utils.fftw3_bdb_utils import save_wisdom, load_wisdom from pyfusion.data.filters import next_nice_number from pyfusion.utils.primefactors import get_fftw3_speed for n in next_nice_number(None):

print(n); get_fftw3_speed(n, flags=[‘FFTW_MEASURE’],direction=’both’,dtype=np.float32)

save_wisdom()

Accepts all pyfftw.FFTW args e.g. planning_timelimit

pyfusion.utils.primefactors.nice_FFT_size(n, max_iterations=None, fft_type='numpy')

Note: Using numpy.fft, it may be better to use the more accurate “fft_time_estimate” above assume cost is Product{N x log(N)} with Ni the factors of n. default max_iterations is ln(N) Works also for below n: just use a negative max_iterations (or -1) Not thoroughly tested. See also next_nice_number in filters.py - more general and simpler

pyfusion.utils.primefactors.nice_FFT_size_above(n, max_iterations=None, fft_type='numpy')

Note: Using numpy.fft, it may be better to use the more accurate “fft_time_estimate” above assume cost is Product{N x log(N)} with Ni the factors of n. default max_iterations is ln(N) Works also for below n: just use a negative max_iterations (or -1) Not thoroughly tested. See also next_nice_number in filters.py - more general and simpler

pyfusion.utils.primefactors.optimise_fit(atimes)

pyfusion.utils.primefactors.primefactors(x, limit=9223372036854775808L)

This is meant to be efficient for fft_time_estimation. 14us for 2**20 298us for 2**20-3

pyfusion.utils.primefactors.primesfrom2to2(n)

pyfusion.utils.primefactors.simple_primefactors(x)

used for testing faster versions 6us to factorize 2**20, 121ms to factorise 2**20-3 (prime)

process_cmd_line_args Module

pyfusion.utils.process_cmd_line_args.process_cmd_line_args()

process_cmd_line_args_code Module

read_csv_data Module

pyfusion.utils.read_csv_data.debug_(debug, level=1, key=None, msg='')

Nice debug function, breaks to debugger if debug>level, or if key is asubstring of the string value in debug. This way, one debug input can control several breakpoints, without too much recoding. Once a possible dedbug point is reasonably OK, its level can be raised, so that it is still accessible, but only at high debug levels. Most calls can be like debug_(debug,3) - where the 3 reduces the priority of this relative to other calls that share the same debug parameter.

What does this mean? Say there are three possible calls to debug in a piece of code where the debug argument is passed in at one point only. Then _debug(debug,3) will break out if debug is very high, whereas debug_(debug) will more easily break out.

key is a string which allows string values of debug to selectively trigger breakout:

e.g key=’lookup’ and debug=’lookup,open’ then execution breaks

pyfusion.utils.read_csv_data.read_csv_data(file_or_list, debug=0, columns=None, header=1, dialect='excel', maxlines=None, openMode='rU')

Reads csv files with given dialect, returns dictionary of arrays of strings If there is a header row, get number of columns from that. about 130k 4 value lines/second (i5/760) Works for LTSPice “export” files, and Mauro’s test data

regulator Module

class pyfusion.utils.regulator.Regulator(maxcpu=0.5)

create a regulator=Regulator(maxcpu) and put regulator.wait() into a loop and it will wait so that max cpu is controlled.

wait()

tests Module

Test code for test code.

class pyfusion.utils.tests.CheckEqualExceptFor

Bases: pyfusion.test.tests.PfTestBase

Test custom object comparison, which allows specified attributed to be ignored

test_equal_except_for()

class pyfusion.utils.tests.CheckPing

Bases: pyfusion.test.tests.PfTestBase

Test the ping utility

testPing()

class pyfusion.utils.tests.Dummy(**kwargs)

time_utils Module

pyfusion.utils.time_utils.utc_ns(s)

return utc corresponding to the string s format 20160302.14:55 e.g. utc_ns([‘20160303.143659’,‘20160303.143661’])

utils Module

pyfusion.utils.utils.compact_str(shot_numbers, max_changing_digits=2, min_run=4, debug=False)

returns a compact string represetation of shots. e.g. [100,101,102,103,109,110,111,120] ==> “100..103,109,110,120” >>> compact_str([10000,10001,10002,10003,10009,10010,10101,10200,11000]) ‘10000,1,2,3,9,10 10101 10200 11000’ >>> compact_str(range(10010,10017),debug=False) ‘10010-10016’

pyfusion.utils.utils.decimate(data, fraction=None, limit=None)

reduce the number of items to a limit or by a fraction

pyfusion.utils.utils.find_last_shot(fun, range=[1, 200000], ex=True, quiet=False)

find the last shot for which fun(shot) does not return an error

Actually more general than this. Only tested for low lim yes, uplim no

>>> def myfun(shot): return(shot <= 12345)
>>> find_last_shot(myfun, quiet=True)
12345

pyfusion.utils.utils.fix2pi_skips(phase, sign='+', around=None, debug=0)

ensure that phase monotonically increases (+) or decreases by adding units of 2Pi - Use modtwopi if you want to keep in that range. !!!! 2013 - fixed problem - loss of two first points! >>> ph = fix2pi_skips([1,8,3,-2]); print(‘, ‘.join([“{0:.3f}”.format(p) for p in ph])) 1.000, 1.717, 3.000, 4.283 >>> ph = fix2pi_skips([4,11,7,1],around=None); print(‘, ‘.join([“{0:.3f}”.format(p) for p in ph])) 4.000, 4.717, 7.000, 7.283 >>> ph = fix2pi_skips([4,11,7,1],around=0); print(‘, ‘.join([“{0:.3f}”.format(p) for p in ph])) -2.283, -1.566, 0.717, 1.000

New method is highly vectorised and much much faster, doesn’t need to know the sign. old method is still here (for now) but not very efficient!

pyfusion.utils.utils.get_local_shot_numbers(partial_name=None, verbose=0, local_path=None, number_posn=[0, 5])

get shots present in path py extracting numbers from matching names This applies to local copies in .npz form. Defaults to local_path=pyfusion.config.get(‘global’,’localdatapath’) partial name defaults to _MP1

pyfusion.utils.utils.modtwopi(x, offset=3.141592653589793)

return an angle in the range of offset +-pi >>> print(“{0:.3f}”.format(modtwopi( 7),offset=3.14)) 0.717 >>> print(“{0:.3f}”.format(modtwopi( -5.12),offset=0)) 1.163

This simple strategy works when the number is near zero +- 2Npi, which is true for calculating the deviation from the cluster centre. does not attempt to make jumps small (use fix2pi_skips for that) but by the correct choice of offset, jumps can be reduced The difference between this and fix2piskips is that this routine always returns within a range of 2pi (around offset).

pyfusion.utils.utils.warn(warning, category=<type 'exceptions.UserWarning'>, stacklevel=2, exception=None)

Similar to warnings.warn, but includes info about the exception. e.g. warn(‘extracting data from shot %d’ % (shot), exception=ex) will print Exception type .., <args>: extraction data from shot 1

Downside compared to print() is that it generates 2-3 lines (including line number of caller) instead of one. Perhaps use for infrequent or more dangerous situations - for “nagging” purposes, print is probably better. Perhaps this should always be used for exception catching.

A minor downside to explicit detail in the message is that multiple messages will be printed if the warning string changes from one error to the next (unless warnings are disabled.)