acquisition Package

acquisition Package

The acquisition package is responsible for fetching data from an experimental database and returning pyfusion data objects. Base classes as well as datasystem-specific sub-packages are provided.

Two classes are involved in obtaining data. An acquisition class (subclass of BaseAcquisition) provides the basic interface to the data source, setting up any connections required. A fetcher class (subclass of BaseDataFetcher) is used to get data from a specified channel and shot number. In general usage, a fetcher class is not handled directly, but via the getdata() method. For example:

>>> import pyfusion
>>> h1 = pyfusion.getDevice('H1')
>>> mirnov_data = h1.acq.getdata(58123, 'H1_mirnov_array_1_coil_1')

Here, h1 is an instance of H1 (the subclass of Device specified in the [Device:H1] section in the configuration file). When instantiated, the device class checks the configuration file for a acquisition class specification, and attaches an instance of the specified acquisition class, here h1.acq (which is a synonym of h1.acquisition). The getdata() method checks for a configuration section (here it is a section named [Diagnostic:H1_mirnov_array_1_coil_1]) with information about the diagnostic including which data fetcher class to use. The data fetcher is then called to fetch and return the data.

base Module

Base classes for pyfusion data acquisition. It is expected that the base classes in this module will not be called explicitly, but rather inherited by subclasses in the acquisition sub-packages.

class pyfusion.acquisition.base.BaseAcquisition(config_name=None, **kwargs)

Bases: object

Base class for datasystem specific acquisition classes.

Parameters:config_name – name of acquisition as specified in configuration file.

On instantiation, the pyfusion configuration is searched for a [Acquisition:config_name] section. The contents of the configuration section are loaded into the object namespace. For example, a configuration section:

[Acquisition:my_custom_acq]
acq_class = pyfusion.acquisition.base.BaseAcquisition
server = my.dataserver.com

will result in the following behaviour:

>>> from pyfusion.acquisition.base import BaseAcquisition
>>> my_acq = BaseAcquisition('my_custom_acq')
>>> print(my_acq.server)
my.dataserver.com

The configuration entries can be overridden with keyword arguments:

>>> my_other_acq = BaseAcquisition('my_custom_acq', server='your.data.net')
>>> print(my_other_acq.server)
your.data.net

getdata(shot, config_name=None, **kwargs)

Get the data and return prescribed subclass of BaseData.

Parameters:

• shot – shot number
• config_name – ?? bdb name of a fetcher class in the configuration file

Returns:

an instance of a subclass of BaseData or BaseDataSet

This method needs to know which data fetcher class to use, if a config_name argument is supplied then the [Diagnostic:config_name] section must exist in the configuration file and contain a data_fetcher class specification, for example:

[Diagnostic:H1_mirnov_array_1_coil_1]
data_fetcher = pyfusion.acquisition.H1.fetch.H1DataFetcher
mds_path = \h1data::top.operations.mirnov:a14_14:input_1
coords_cylindrical = 1.114, 0.7732, 0.355
coord_transform = H1_mirnov

If a data_fetcher keyword argument is supplied, it overrides the configuration file specification.

The fetcher class is instantiated, including any supplied keyword arguments, and the result of the fetch method of the fetcher class is returned.

class pyfusion.acquisition.base.BaseDataFetcher(acq, shot, config_name=None, **kwargs)

Bases: object

Base class providing interface for fetching data from an experimental database.

Parameters:

• acq – in instance of a subclass of BaseAcquisition
• shot – shot number
• config_name – name of a Diagnostic configuration section.

It is expected that subclasses of BaseDataFetcher will be called via the getdata() method, which calls the data fetcher’s fetch() method.

do_fetch()

Actually fetches the data, using the environment set up by setup()

Returns:an instance of a subclass of BaseData or BaseDataSet

Although BaseDataFetcher.do_fetch() does not return any data object itself, it is expected that a do_fetch() method on a subclass of BaseDataFetcher will.

error_info(step=None)

return specific information about error to aid interpretation - e.g for mds, path The dummy return should be replaced in the specific routines

fetch()

Always use this to fetch the data, so that setup() and pulldown() are used to setup and pull down the environmet used by do_fetch().

Returns:the instance of a subclass of BaseData or BaseDataSet returned by do_fetch()

find_valid_for_shot()

Determine if this diag definition or modified diag is valid for this shot

pulldown()

Called by fetch() after retrieving the data.

setup()

Called by fetch() before retrieving the data. setup() ideally does sufficient of the preliminaries for fetch so that their is enough information in self. for a useful error report if fetch fails, while avoiding exceptions wherever possible (during setup()).

class pyfusion.acquisition.base.MultiChannelFetcher(acq, shot, config_name=None, **kwargs)

Bases: pyfusion.acquisition.base.BaseDataFetcher

Fetch data from a diagnostic with multiple timeseries channels.

This fetcher requres a multichannel configuration section such as:

[Diagnostic:H1_mirnov_array_1]
data_fetcher = pyfusion.acquisition.base.MultiChannelFetcher
channel_1 = H1_mirnov_array_1_coil_1
channel_2 = H1_mirnov_array_1_coil_2
channel_3 = H1_mirnov_array_1_coil_3
channel_4 = H1_mirnov_array_1_coil_4

The channel names must be channel_ followed by an integer, and the channel values must correspond to other configuration sections (for example [Diagnostic:H1_mirnov_array_1_coil_1], [Diagnostic:H1_mirnov_array_1_coil_1], etc) which each return a single channel instance of TimeseriesData.

fetch()

Fetch each channel and combine into a multichannel instance of TimeseriesData.

Return type:TimeseriesData

ordered_channel_names()

Get an ordered list of the channel names in the diagnostic

Return type:list

pyfusion.acquisition.base.newloadv3(filename, verbose=1)

This the the version that works in python 3, but can’t handle Nans Intended to replace load() in numpy counterpart is data/savez_compress.py

pyfusion.acquisition.base.try_fetch_local(input_data, bare_chan)

return data if in the local cache, otherwise None doesn’t work for single channel HJ data. sgn (not gain) be only be used at the single channel base/fetch level

process_chirps Module

These routines calculate df/dt for points in the same shot which have been identified as belonging to a mode group pseudocode:

for all shots

for all mode groups (N,M)

link points

saving back into the original data set via the indx v1: does 250k shots in ~ 1min. uses maxd and dfdtunit, bool for islead v2: islead is now the count of points linked, minlen, tidy output

9/2013 run -i pyfusion/examples/process_chirps.py Ns=None debug=0 Ms=[None] shots=unique(shot) verbose=5 11/2013 killed bug that prevent correct write back (used indx, should be inds[tord]) Also try maxd ~ 10 (5 missed fast chirps). However it is becoming clear that separate time and freq scale units should replace dfdtunit.

pyfusion.acquisition.process_chirps.plotdfdt(dd, inds, join=False, **kwargs)

pyfusion.acquisition.process_chirps.process_chirps(dd, shots=None, Ns=None, Ms=None, maxd=4, dfdtunit=1000, minlen=2, plot=0, hold=0, clear_dfdt=False, debug=0, verbose=0)

M and N default to all values present >>>

pyfusion.acquisition.process_chirps.process_chirps_m_shot(t_mid, freq, indx, maxd=4, dfdtunit=1000, minlen=2, plot=0, debug=0)

>>> indx = np.array([1,2,3,4]); freq = np.array([5,4,2,1])
>>> t_mid = np.array([1,2,3,10])
>>> process_chirps_m_shot( t_mid, freq, indx, maxd=4, dfdtunit=1, debug=0)
3 connected of 4 chirping,
(array([3, 0, 0, 0]), array([ 1,  1,  1, -1]), array([ -1.,  -2.,  nan,  nan], dtype=float32))

read_text_pyfusion Module

pyfusion.acquisition.read_text_pyfusion.find_data(file, target, skip=0, recursion_level=0, debug=0)

find the last line number which contains the string (regexp) target work around a “feature” of loadtxt, which ignores empty lines while reading data, but counts them while skipping.

2016 - this bug was not fixed up until Feb2016, failed if there were blank lines before the Shot line - fixed by double loop (see final_tries)

Really should consider rewriting without loadtxt - this could also allow us to avoid a lot of the b’ ‘ stuff

Using loadtxt at least for now: This way, everything goes through one reader (instead of gzip.open), simplifying, and allowing caching. The recursive part could be handled more nicely. about 0.14 secs to read 3500 lines, 0.4 if after 11000 lines of text

pyfusion.acquisition.read_text_pyfusion.merge_ds(ds_list, comment_list=[], old_dd=None, debug=True, force=False)

Take a list of structured arrays, and merge into one Adding to an existing dd is not fully tested - may be memory intensive and does not check for keys being different in the ds and dd

pyfusion.acquisition.read_text_pyfusion.read_text_pyfusion(files, target='^Shot .*', ph_dtype=None, plot=False, ms=100, hold=0, debug=0, quiet=1, maxcpu=1, exception=<type 'exceptions.Exception'>)

Accepts a file or a list of files, returns a list of structured arrays See merge ds_list to merge and convert types (float -> pyfusion.prec_med

tests Module

Test code for data acquisition.

class pyfusion.acquisition.tests.CheckAcquisitionArgs

Bases: pyfusion.test.tests.PfTestBase

Make sure we get the same result if we use config or kwargs

testAcqAttrsConfig()

Check that config, kwarg attributes are correctly attached to object.

If config is supplied, load config before kwargs.

testAcqAttrsConfigKwargs()

Check that config, kwarg attributes are correctly attached to object.

If config is supplied, load config before kwargs.

testAcqAttrsKwargs()

Check that config, kwarg attributes are correctly attached to object.

If config is supplied, load config before kwargs.

testEqualityConfigOrArgs()

Check that config and kwarg instantiated Acquisition classes are same.

class pyfusion.acquisition.tests.CheckAcquisitionModules

Bases: pyfusion.test.tests.PfTestBase

Check for existence of acquisition modules.

check_module(module_name)

testModules()

class pyfusion.acquisition.tests.CheckDataFetchers

Bases: pyfusion.test.tests.PfTestBase

test DataFetcher subclasses for fake data acquisition.

testDummyFetcher()

testFetcherShotArg()

test_setup_pulldown()

class pyfusion.acquisition.tests.CheckGetAcquisition

Bases: pyfusion.test.tests.PfTestBase

test getAcquisition function.

test_get_acquistion()

class pyfusion.acquisition.tests.DummyFetcher(*args, **kwargs)

Bases: pyfusion.acquisition.base.BaseDataFetcher

do_fetch()

pulldown()

setup()

utils Module

Utilities for malipulating acquisition objects.

pyfusion.acquisition.utils.getAcquisition(acq_name)

Return the an instance of the acquisition class specified by acq_class in the [Acquisition:acq_name] section of the configuration file.

Parameters:acq_name – name of the acquisition configuration section. Returns:instance of the acquisition class specified by acq_class in the [Acquisition:acq_name] section of the configuration file

pyfusion.acquisition.utils.get_acq_from_config(acq_name)

Return the acquisition class specified by acq_class in the [Acquisition:acq_name] section of the configuration file.

Parameters:acq_name – name of the acquisition configuration section. Returns:acquisition class specified by acq_class in the [Acquisition:acq_name] section of the configuration file

Subpackages

• DSV Package
  • DSV Package
  • acq Module
  • fetch Module
  • test Module
• FakeData Package
  • FakeData Package
  • acq Module
  • fetch Module
  • tests Module
• H1 Package
  • H1 Package
  • fetch Module
  • get_basic_diagnostics Module
• HeliotronJ Package
  • HeliotronJ Package
  • acq Module
  • fetch Module
  • get_basic_diagnostics Module
  • get_hj_modules Module
  • make_static_param_db Module
  • manual_recompile Module
• LHD Package
  • LHD Package
  • LHD_utils Module
  • acq Module
  • evaluate_other Module
  • fetch Module
  • fetch_local Module
  • fetch_local_tmp Module
  • get_basic_diagnostics Module
  • import_LHD_summary_csv Module
  • make_summary_egdb Module
  • read_igetfile Module
  • tests Module
• MDSPlus Package
  • MDSPlus Package
    • Local data access
    • Thick client access
    • Thin client access
    • HTTP web service access
    • How Pyfusion chooses the access mode
  • acq Module
  • fetch Module
  • h1ds Module
  • tests Module
• TJII Package
  • TJII Package
  • acq Module
  • fetch Module
• W7X Package
  • W7X Package
  • acq Module
  • fetch Module
  • get_shot_info Module
  • get_shot_list Module
  • mag_config Module
  • puff_db Module
  • scrape_archive Module
• binary Package
  • binary Package
  • acq Module
  • fetch Module
  • test Module