Codestin Search App

JGoodrichBNL · 2026-08-27T21:01:03Z

Minimalistic changes to CHX's collection profile to remove Databroker API and replace with Tiled throughout.

Branch has been tested at beamline; common functions such as series(), ps(), etc. work without issue. Moved current main to branch "databroker_fallback" if issues are encountered during early operations in 2026-3.

… and use Tiled API throughout.

…aces it was used to new functions instead.

Eugene M. (genematx) · 2026-08-27T21:57:46Z

+db = tiled_reading_client
+
+
+def get_run_start(run):


can use run.start (and run.stop) rightaway

I like 2. it addresses my comment for the optional startup py files

Eugene M. (genematx) · 2026-08-27T21:58:57Z

+
+
+def get_run_data(run, stream_name="primary"):
+    return run[stream_name]["data"]


with the new Tiled layout, you shouldn't need ["data"], but maybe I'm misinterpreting the intention here

Eugene M. (genematx) · 2026-08-27T22:02:18Z

+    ds = get_run_data(run, stream_name)
+    keys = list(ds.keys()) if fields is None else fields
+    data = {}
+    for key in keys:


might be faster to read the entire dataset as xarray (possibly filtering out only the columns you need), ds.read(variables=fields)

if you do a Eugene M. (@genematx) suggests, then you can just convert the xarray data broker returns directly to a Dataframe with .to_dataframe()

this worked beautifully for scalar data and for 1D arrays within each "bluesky" point.

If this functions fails on >2D, then I think embracing that and handling the exception makes good sense. One is getting a pandas data frame for 1D fitting or pandas builtins

Eugene M. (genematx) · 2026-08-27T22:04:55Z

    if auto_compression:
        try:
-            uid_add=tiled_reading_client[-1]['start']['uid']
+            uid_add=get_run_start(tiled_reading_client[-1])['uid']


[-1] could be slow (will become faster eventually, but the work is still in progress). A useful workaround is to find the last key/uuid, and then use it to index the container, e.g.

key = catalog.keys().last() run = catalog[key]

instead of run = catalog[-1].

But in this specific case, maybe all you need is just the key...

Andi Barbour (ambarb)

I don't need to re-review. happy for you to take the suggestions you like and leave the ones you don't

Andi Barbour (ambarb) · 2026-08-28T16:47:24Z

            check_current_beam(bpm_int_threshold=-1,wait_for_beam=True)
            print('series(expt=.1,imnum=200,....)')
-            waiting_for_data = check_past_data(db[-1].start['uid'],bpm_int_threshold=-1,fraction=.05,verbose=True)
+            waiting_for_data = check_past_data(get_run_start(db[-1])['uid'],bpm_int_threshold=-1,fraction=.05,verbose=True)


where is get_run_start() defined or imported? Exists in name space, I assume, but unclear to me

Andi Barbour (ambarb) · 2026-08-28T16:48:40Z

+db = tiled_reading_client
+
+
+def get_run_start(run):


I like 2. it addresses my comment for the optional startup py files

Andi Barbour (ambarb) · 2026-08-28T16:53:19Z

+    return list(get_run_data(run, stream_name).keys())
+
+
+def get_table(run, fields=None, stream_name="primary"):


I prefer a hierarchical approach: run, stream, fields
but nit-picking here.

Andi Barbour (ambarb) · 2026-08-28T16:55:04Z

+    ds = get_run_data(run, stream_name)
+    keys = list(ds.keys()) if fields is None else fields
+    data = {}
+    for key in keys:


if you do a Eugene M. (@genematx) suggests, then you can just convert the xarray data broker returns directly to a Dataframe with .to_dataframe()

this worked beautifully for scalar data and for 1D arrays within each "bluesky" point.

If this functions fails on >2D, then I think embracing that and handling the exception makes good sense. One is getting a pandas data frame for 1D fitting or pandas builtins

Andi Barbour (ambarb) · 2026-08-28T17:03:25Z

    exclude_list = ['uid','scan_id', 'time'] # we should not overwrite these
-    h=db[uid].v2
+    h=db[uid]
+    start = get_run_start(h)


can we use something like start_md? start is arbitrary and depends on context. I had to scroll up and remind myself what get_run_start() did. Instead I wonder if just using the tiled functions provided is more streamlined, given that is the thing that should work no matter the beamline.

Andi Barbour (ambarb) · 2026-08-28T17:04:29Z

    hdr = tiled_reading_client[uid]
-    keys = [k for k, v in hdr.descriptors[0]['data_keys'].items()     if 'external' in v]
-    det = keys[0]    
+    det = get_run_start(hdr)['detectors'][0]


this assumes the first detector in the list is the one you want. I think it is better to handle this with a variable. What if one is using multiple detectors?

Andi Barbour (ambarb) · 2026-08-28T20:57:37Z

+def get_sid_filenames(header): 
    """YG. Dev Jan, 2016
    ---------------- DEPRECATED -----------------
    Get a bluesky scan_id, unique_id, filename by giveing uid


fix typo while here?

if the function didn't work, can we just throw a deprecation error, and make a new function called get_scan_uid_filenames()?

Andi Barbour (ambarb) · 2026-08-28T21:02:47Z

-        filepaths.extend(db.reg.get_file_list(uid, datum_kwarg_gen))
-    return header.start['scan_id'],  header.start['uid'], filepaths
+    start = get_run_start(header)
+    return start['scan_id'],  start['uid'], []


this function (as the name implies), originally tried to give the file names for specific scan uids. The function no longer does that so why are use using it?

It's unclear to me which filenames , looks like the detector files names. There is a better way to do that in tiled. Should we leave an optional variable to retrieve the file names for a specific detector do we want it for all assets only?

Andi Barbour (ambarb) · 2026-08-28T21:03:39Z

+        motor, = start['motors']
+        data_keys = get_fields(header)
        for key in data_keys:
            if key.endswith('stats1_total'):


does it have to be stats1_total? can we make it more flexible?

maybe it doesn't matter if this code isn't being used any more - so happy if this is a "won't do" or a "won't do" because plotting and fitting are going to be refactored later.

Andi Barbour (ambarb) · 2026-08-28T21:07:24Z

-get_table = db.get_table
-from matplotlib import pyplot as pltfrom
+from PIL import Image
+from matplotlib import pyplot as pltfrom


from matplotlib import pyplot as pltfrom seems wrong. should it not be from matplotlib import pyplot as plt

maybe that is why there is the imports inside many of the functions that are
from matplotlib import pyplot as plt

Copilot

🟡 Changes recommended

There are confirmed runtime issues and breaking behaviors in updated utilities/tests (notably db[-1] usage and get_scan() resolution) that should be fixed before approval.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

This PR migrates the CHX profile collection from the Databroker API to direct Tiled usage, aiming to preserve existing beamline scientist workflows while removing the Databroker dependency.

Changes:

Replace the Databroker Broker(...) wrapper with a Tiled reading client and add lightweight compatibility helpers (get_fields, get_table, get_images).
Update startup macros/utilities to use Tiled run access patterns (notably db.keys().last() for “latest run”).
Remove the databroker dependency from the pixi environment and adjust acceptance-test scripts accordingly.

File summaries

File	Description
startup/00-base.py	Switch `db` to the Tiled catalog and add helper functions to replace common Databroker conveniences.
startup/30-user.py	Update metadata and image-field selection logic to use Tiled APIs and keys.
startup/37-database.py	Remove Databroker imports and update “latest UID” retrieval to Tiled keys.
startup/37-database2.py	Same as above for the alternate database macro file.
startup/39_db.py	Add a new Tiled-based filename/asset helper and deprecate the old helper.
startup/91-run-browser-gui.py	Rework plotting logic to use `get_table`/`get_fields` rather than Databroker processing.
startup/95-utilities.py	Replace Databroker calls with Tiled helpers across scan/plot/export utilities.
startup/96-util_funcs.py	Remove Databroker-specific imports and route through shared helpers.
startup/97_HDM.py	Update last-run retrieval to Tiled key semantics.
startup/99-bluesky.py	Update last-run table retrieval to Tiled key semantics and adjust comments.
optional_startup/lutz_macros/check_beam.py	Update “latest run” access to use `db.keys().last()`.
acceptance_tests/CHX_minitest_052026.py	Update example image reads to use `get_images` (still needs Tiled-latest fixups).
acceptance_tests/CHX_acceptancetest_041422.py	Update image reads to `get_images` (still needs Tiled-latest fixups).
pixi.toml	Drop the `databroker` dependency from the environment.

Review details

Suppressed comments (12)

startup/95-utilities.py:460

get_scan() currently assumes db[...] is keyed by the provided scan_id, but with db set to the Tiled catalog this will not handle common cases like -1/'-1' (latest) and may not resolve integer scan_id values. This breaks get_data() (which delegates to get_scan) for typical interactive usage.

def get_scan(scan_id, debug=False):
    """Get scan from Tiled using provided scan id.
from Maksim
    :param scan_id: scan id from bluesky.
    :param debug: a debug flag.
    :return: a tuple of scan and timestamp values.
    """
    scan = db[scan_id]
    #t = datetime.datetime.fromtimestamp(scan.start['time']).strftime('%Y-%m-%d %H:%M:%S')
    #t = dtt.datetime.fromtimestamp(scan.start['time']).strftime('%Y-%m-%d %H:%M:%S')
    t='N.A. conflicting with other macro'
    if debug:
        print(scan)
    print('Scan ID: {}  Timestamp: {}'.format(scan_id, t))
    return scan, t

startup/95-utilities.py:760

In the file=='ia' branch, file_path is never set (the Tkinter code is commented out), but description=file_path will still execute and raise NameError. Since interactive selection is not supported in this environment, fail fast with a clear exception.

    if file=='ia':          # open file dialog
        print('this would open a file input dialog IF Tkinter was available in the $%^& python environment as it used to')
        #root = Tkinter.Tk()
        #root.withdraw()
        #file_path = tkFileDialog.askopenfilename()
        description=file_path

acceptance_tests/CHX_acceptancetest_041422.py:20

db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client), so db[-1] will not return the most recent run. Use db[db.keys().last()] to fetch the latest run before calling get_images().

img = get_images(db[-1], 'eiger1m_single_image');

acceptance_tests/CHX_acceptancetest_041422.py:31

db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client), so db[-1] will not return the most recent run. Use db[db.keys().last()] to fetch the latest run before calling get_images().

img = get_images(db[-1], 'eiger4m_single_image');

acceptance_tests/CHX_acceptancetest_041422.py:37

db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client), so db[-1] will not return the most recent run. Use db[db.keys().last()] to fetch the latest run before calling get_images().

img = get_images(db[-1], 'eiger4m_single_image');

acceptance_tests/CHX_acceptancetest_041422.py:56

db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client), so db[-1] will not return the most recent run. Use db[db.keys().last()] to fetch the latest run before calling get_images().

img = get_images(db[-1], 'eiger500K_single_image');

acceptance_tests/CHX_minitest_052026.py:20

This commented example still uses db[-1], but db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client) so negative indexing is no longer valid. Update the example to avoid copy/paste failures when the image read is re-enabled.

#img = get_images(db[-1], 'eiger1m_single_image');

acceptance_tests/CHX_minitest_052026.py:32

This commented example still uses db[-1], but db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client) so negative indexing is no longer valid. Update the example to avoid copy/paste failures when the image read is re-enabled.

#img = get_images(db[-1], 'eiger4m_single_image');

acceptance_tests/CHX_minitest_052026.py:39

This commented example still uses db[-1], but db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client) so negative indexing is no longer valid. Update the example to avoid copy/paste failures when the image read is re-enabled.

#img = get_images(db[-1], 'eiger4m_single_image');

acceptance_tests/CHX_minitest_052026.py:53

This commented example still uses db[-1], but db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client) so negative indexing is no longer valid. Update the example to avoid copy/paste failures when the image read is re-enabled.

#img = get_images(db[-1], 'eiger500K_single_image');

acceptance_tests/CHX_minitest_052026.py:60

This commented example still uses db[-1], but db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client) so negative indexing is no longer valid. Update the example to avoid copy/paste failures when the image read is re-enabled.

#img = get_images(db[-1], 'eiger500K_single_image');

acceptance_tests/CHX_acceptancetest_041422.py:50

db is now a Tiled catalog (startup/00-base.py sets db = tiled_reading_client), so db[-1] will not return the most recent run. Use db[db.keys().last()] to fetch the latest run before calling get_images().

img = get_images(db[-1], 'eiger500K_single_image');

Files reviewed: 14/14 changed files
Comments generated: 4
Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

 RE(count([eiger1m_single]),Measurement='Deployment test')

-img = db[-1].xarray_dask()['eiger1m_single_image'];
+img = get_images(db[-1], 'eiger1m_single_image');


 def get_sid_filenames(header):
-    """YG. Dev Jan, 2016
-    ---------------- DEPRECATED -----------------
-    Get a bluesky scan_id, unique_id, filename by giveing uid
-
+    """Deprecated; use :func:`get_scan_uid_filenames` instead."""
+    raise DeprecationWarning(
+        "get_sid_filenames() never returned filenames reliably; "
+        "use get_scan_uid_filenames() instead"
+    )


 RE(count([eiger1m_single]),Measurement='Deployment test')

-#img = db[-1].xarray_dask()['eiger1m_single_image'];
+#img = get_images(db[-1], 'eiger1m_single_image');


    function to read energy scan file and determine offset correction
    calling sequence: E_calibration(file,Edge='Cu',xtal='Si111cryo',B_off=0)
-    file: path/filename of experimental data; 'ia' opens interactive dialog; file can be databrooker object, e.g. file=db[-1] t process data from last scan
+    file: path/filename of experimental data; 'ia' opens interactive dialog; file can be Tiled run object, e.g. file=db[-1] to process data from last scan


JGoodrichBNL added 3 commits August 25, 2026 17:04

Attempt to remove Databroker API references throughout the repository…

7fb374a

… and use Tiled API throughout.

Removed descriptors utility function which didn't work and updated pl…

c87642f

…aces it was used to new functions instead.

Added note to depreciated function.

5b0441a

JGoodrichBNL requested review from Andi Barbour (ambarb) and Eugene M. (genematx) and a lite review from Copilot and removed request for Copilot August 27, 2026 21:01

Copilot started reviewing on behalf of JGoodrichBNL August 27, 2026 21:01 View session

Eugene M. (genematx) reviewed Aug 27, 2026

View reviewed changes

Andi Barbour (ambarb) approved these changes Aug 28, 2026

View reviewed changes

Address Tiled migration review feedback

8d4a513

Copilot AI lite review requested due to automatic review settings September 9, 2026 18:54

Copilot started reviewing on behalf of JGoodrichBNL September 9, 2026 18:54 View session

Copilot AI reviewed Sep 9, 2026

View reviewed changes



		def get_run_data(run, stream_name="primary"):
		return run[stream_name]["data"]

		return list(get_run_data(run, stream_name).keys())


		def get_table(run, fields=None, stream_name="primary"):

Uh oh!

Conversation

JGoodrichBNL commented Aug 27, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andi Barbour (ambarb) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Andi Barbour (ambarb) Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Suppressed comments (12)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Andi Barbour (ambarb) Aug 28, 2026 •

edited

Loading