Puncta Module
Puncta Extraction
Puncta extraction module facilitates the extraction of puncta—distinct fluorescent spots indicative of molecular targets—from expansion microscopy data. It provides functions for processing this data on both CPUs and GPUs, enabling flexible and high-throughput analysis. The extraction process identifies puncta coordinates and saves them for further analysis.
- exm.puncta.extract.calculate_coords_cpu(args, tasks_queue, queue_lock)[source]
Extracts puncta from volumes included in the task queue using CPU and saves their locations to a .pkl file.
- exm.puncta.extract.calculate_coords_gpu(args, tasks_queue, device, lock, queue_lock)[source]
Extracts puncta from volumes included in the task queue and saves their locations to a .pkl file using GPU acceleration.
- Parameters:
args (Args) – Configuration options. This should be an instance of the Args class.
tasks_queue (Queue[Tuple[int, int]]) – A queue of tasks, where each task is a (code, fov) pair.
device (int) – GPU device ID.
lock (Lock) – A multiprocessing lock instance to avoid race condition when processes accessing the GPU.
queue_lock (Lock) – A multiprocessing lock instance to avoid race condition when processes accessing the task queue.
- Return type:
None
- exm.puncta.extract.extract(args, code_fov_pairs, use_gpu=False, num_gpu=3, num_cpu=3)[source]
Runs the extraction process (calculate_coords_cpu or calculate_coords_gpu) for all codes and FOVs specified in code_fov_pairs.
- Parameters:
args (Args) – Configuration options. This should be an instance of the Args class.
code_fov_pairs (List[Tuple[int, int]]) – A list of tuples, where each tuple is a (code, fov) pair.
use_gpu (bool) – Whether or not to enable GPU processing. Default is False.
num_gpu (int) – Number of GPUs to use for processing. Default is 3.
num_cpu (Optional[int]) – Number of CPUs to use for processing. Default is None which means it will use a quarter of available CPUs.
- Return type:
None
- exm.puncta.extract.puncta_extraction_cpu(args, tasks_queue, num_cpu)[source]
Wrapper around calculate_coords_cpu to enable parallel processing on the CPU.
Puncta Consolidation
Puncta consolidation module is designed to refine puncta detection by eliminating redundancies across different channels and codes. It processes the data to merge closely located puncta into a single entity based on a distance threshold, which helps in reducing noise and improving the accuracy of the puncta detection process.
- exm.puncta.consolidate.consolidate_channels(args, code_fov_pairs, num_cpu=None)[source]
Wrapper around consolidate_channels_function to enable parallel processing.
- Parameters:
args (Args) – Configuration options. This should be an instance of the Args class.
code_fov_pairs (List[Tuple[int, int]]) – A list of tuples, where each tuple is a (code, fov) pair.
num_cpu (int, optional) – Number of CPUs to use for processing. If not provided, defaults to a quarter of the available CPUs.
- Return type:
None
- exm.puncta.consolidate.consolidate_channels_function(args, tasks_queue, q_lock)[source]
Consolidates (removes duplicate puncta) across channels using distance thresholding, called by consolidate_channels.
- Parameters:
args (Any) – Configuration options. This should be an instance of a configuration class or similar structure.
tasks_queue (multiprocessing.Queue[Tuple[int, int]]) – Queue containing tuples of field of view (FOV) and code to be processed.
q_lock (multiprocessing.Lock) – Lock for ensuring process-safe operations on the tasks queue.
- Return type:
None
- exm.puncta.consolidate.consolidate_codes(args, fov_list, num_cpu=None)[source]
Wrapper around consolidate_codes_function to enable parallel processing.
- exm.puncta.consolidate.consolidate_codes_function(args, tasks_queue, q_lock)[source]
Reads in the locations of the puncta from a specified field of view (FOV), then uses distance thresholding to consolidate (i.e., remove duplicate puncta) across codes.
- Parameters:
args (Any) – Configuration options. This should be an instance of a configuration class or similar structure.
tasks_queue (Queue) – Queue containing tasks to be processed.
q_lock (multiprocessing.Lock) – Lock for ensuring process-safe operations on the tasks queue.
- Return type:
None
Puncta Inspection
This Puncta visualization and inspection module assists users in visualizing and inspecting puncta, which are typically bright spots representing the presence of specific molecules or structures in microscopy images. The module contains functions to visualize raw or processed image data within predefined regions of interest (ROIs) and to confirm the accuracy of puncta detection algorithms.
- exm.puncta.inspect.in_region(coord, ROI_min, ROI_max)[source]
Given a coordinate location and lower and upper bounds for a volume chunk (region), returns whether or not the coordinate is inside the chunk.
- exm.puncta.inspect.inspect_across_rounds_plotly(args, fov, ROI_min, ROI_max, spacer=20, export_file_name=None)[source]
Plots puncta across rounds (for all codes) using Plotly.
- Parameters:
args (args.Args) – configuration options.
fov (int) – the field of fiew of the volume chunk to be returned.
ROI_min (list) – minimum coordinates of the volume chunk to display. Expects coordinates in the format of \([z, y, x]\).
ROI_max (list) – maximum coordinates of the volume chunk to display. Expects coordinates in the format of \([z, y, x]\).
spacer (int) – scaling factor to use for z-spacing. Default:
20export_file_name (str) – name of the file to be exported. Default:
None
- exm.puncta.inspect.inspect_between_rounds_plotly(args, fov, code1, code2, ROI_min, ROI_max, spacer=40, export_file_name=None)[source]
Plots puncta across rounds (for two specified codes) using Plotly.
- Parameters:
args (args.Args) – configuration options.
fov (int) – the field of fiew of the volume chunk to be returned.
code1 (str) – name of the first code to include in comparison.
code2 (str) – name of the first code to include in comparison.
ROI_min (list) – minimum coordinates of the volume chunk to display. Expects coordinates in the format of \([z, y, x]\).
ROI_max (list) – maximum coordinates of the volume chunk to display. Expects coordinates in the format of \([z, y, x]\).
spacer (int) – scaling factor to use for z-spacing. Default:
40export_file_name (str) – name of the file to be exported. Default:
None
- exm.puncta.inspect.inspect_localmaximum_matplotlib(args, fov, code, ROI_min, ROI_max, vmax=500)[source]
Plots middle slice of each channel for a specific fov/code using Matplotlib.
- Parameters:
args (args.Args) – configuration options.
fov (int) – the field of fiew of the volume chunk to be returned.
code (int) – the code of the volume chunk to be returned.
channel (int) – the channel of the volume chunk to be returned.
ROI_min (list) – minimum coordinates of the volume chunk. Expects coordinates in the format of \([z, y, x]\).
ROI_max (list) – maximum coordinates of the volume chunk. Expects coordinates in the format of \([z, y, x]\).
vmax (int) – maximum pixel intensity to display. Default:
500
- exm.puncta.inspect.inspect_localmaximum_plotly(args, fov, code, channel, ROI_min, ROI_max, export_file_name=None)[source]
Plots identified puncta for a specific fov/code/channel using Plotly.
- Parameters:
args (args.Args) – configuration options.
fov (int) – the field of fiew of the volume chunk to be returned.
code (int) – the code of the volume chunk to be returned.
channel (int) – the channel of the volume chunk to be returned.
ROI_min (list) – minimum coordinates of the volume chunk. Expects coordinates in the format of \([z, y, x]\).
ROI_max (list) – maximum coordinates of the volume chunk. Expects coordinates in the format of \([z, y, x]\).
export_file_name (str) – name of the file to be exported. Default:
None
- exm.puncta.inspect.inspect_puncta_ROI_matplotlib(args, fov, code, position, center_dist=40)[source]
Plots identified puncta for a specific fov/code using Matplotlib. Assumes the puncta have already been consolidated accross channels.
- Parameters:
args (args.Args) – configuration options.
fov (int) – the field of fiew of the volume chunk to be returned.
code (int) – the code of the volume chunk to be returned.
position (list) – the center point of the region that should be visualized. Expects position in the format of \([z, y, x]\).
center_dist (int) – distance from the center that should be viewable. Default:
40
- exm.puncta.inspect.inspect_puncta_ROI_plotly(args, fov, position, c_list=[0, 1, 2, 3], center_dist=40, spacer=40, export_file_name=False)[source]
Plots identified puncta for a specific fov/code using Plotly. Assumes the puncta have already been consolidated accross channels.
- Parameters:
args (args.Args) – configuration options.
fov (int) – the field of fiew of the volume chunk to be returned.
position (list) – the center point of the region that should be visualized. Expects position in the format of \([z, y, x]\).
c_list (list) – the codes to include in the viaulization.
center_dist (int) – distance from the center that should be viewable. Default:
40spacer (int) – scaling factor to use for z-spacing. Default:
40export_file_name (str) – name of the file to be exported. Default:
None
- exm.puncta.inspect.inspect_puncta_improvement_matplotlib(args, fov, puncta_index, option='final', center_dist=40, save=False, missing_code=0)[source]
Visualizes puncta improvement using Matplotlib. The function generates a detailed plot of the region of interest (ROI) around a given puncta and shows changes in the puncta position over different rounds of image acquisition. The function supports visualization of missing code if provided.
- Parameters:
args (args.Args instance) – Configuration options, including methods for retrieving puncta and images.
fov (int) – The field of view (fov) to consider.
puncta_index (int) – The index of the puncta to start the search from.
option (str) – Option for puncta visualization, should be either ‘initial’, ‘intermediate’, or ‘final’. Default is ‘final’.
center_dist (int) – The distance to the center of the ROI. Default is 40.
save (bool) – Whether to save the generated plot or not. If set to False, the plot will be displayed. Default is False.
missing_code (int) – The missing code to consider in the visualization. Default is 0.
- Returns:
None
- exm.puncta.inspect.inspect_puncta_individual_matplotlib(args, fov, puncta_index, center_dist=40)[source]
Plots specified puncta using Matplotlib. Assumes the puncta have already been consolidated accross channels.
- exm.puncta.inspect.inspect_puncta_individual_plotly(args, fov, puncta, center_dist=40, spacer=40, save=False)[source]
Visualizes specified puncta in a 3D space using Plotly. Assumes the puncta have already been consolidated across channels.
- Parameters:
args (args.Args) – configuration options.
fov (int) – Identifier of the specific region in the image dataset, the field of view of the volume chunk to be returned.
puncta (dict) – The puncta to visualize. Should contain information about the puncta’s position in the 3D space and its index.
center_dist (int) – Distance from the center of the puncta that should be viewable in the plot. Default:
40spacer (int) – Scaling factor used for z-spacing to separate different rounds of imaging in the 3D plot. Default:
40save (bool) – If True, the plot will be saved as an HTML file in the directory specified by args.puncta_path. Default:
False
This function generates an interactive 3D scatter plot using Plotly, where each puncta is represented as a point in the 3D space. The plot also includes slices of images from different rounds of imaging, providing a contextual understanding of puncta positioning.
Note: The function assumes that the puncta have already been consolidated across different channels and rounds of imaging.
- exm.puncta.inspect.inspect_raw_matplotlib(args, fov, code, channel, ROI_min, ROI_max, vmax=500, mode='raw')[source]
Plots the middle slice of a specified volume chunk using Matplotlib.
- Parameters:
args (args.Args) – configuration options.
fov (int) – the field of fiew of the volume chunk to be returned.
code (int) – the code of the volume chunk to be returned.
channel (int) – the channel of the volume chunk to be returned.
ROI_min (list) – minimum coordinates of the volume chunk. Expects coordinates in the format of \([z, y, x]\).
ROI_max (lsit) – maximum coordinates of the volume chunk. Expects coordinates in the format of \([z, y, x]\).
vmax (int) – maximum pixel intensity to display. Default:
500mode (str) – expects ‘raw’ or ‘blur’. ‘raw’ plots the images as-is, ‘blur’ applies Gaussian blurring before plotting. Default:
'raw'
- exm.puncta.inspect.inspect_raw_plotly(args, fov, code, channel, ROI_min, ROI_max, vmax=500, mode='raw', export_file_name=False)[source]
Plots the middle slice of a specified volume chunk using Plotly.
- Parameters:
args (args.Args) – configuration options.
fov (int) – the field of fiew of the volume chunk to be returned.
code (int) – the code of the volume chunk to be returned.
channel (int) – the channel of the volume chunk to be returned.
ROI_min (list) – minimum coordinates of the volume chunk. Expects coordinates in the format of \([z, y, x]\).
ROI_max (list) – maximum coordinates of the volume chunk. Expects coordinates in the format of \([z, y, x]\).
vmax (int) – maximum pixel intensity to display. Default:
500mode (str) – expects ‘raw’ or ‘blur’. ‘raw’ plots the images as-is, ‘blur’ applies Gaussian blurring before plotting. Default:
'raw'export_file_name (str) – name of the file to be exported. Default:
False
- exm.puncta.inspect.plot_genes_global(args, tileset, zslice, gene_list=['All'], title='Global Genes', mask_path=None, improved=False, save=False)[source]
Plots the global distribution of specified genes. Each gene is represented as a scatter plot on a 2D slice of the global space. If ‘All’ is specified in gene_list, all possible genes will be plotted. Optionally, a mask can be applied, and only genes within the mask will be plotted.
- Parameters:
args (args.Args instance) – Configuration options.
tileset (exm.stitching.tileset.Tileset) – The tileset object from which slices will be displayed.
zslice (int) – The Z coordinate for the slice to be displayed.
gene_list (list) – List of genes to be plotted. If [‘All’] is passed, all possible genes will be plotted. Defaults to [‘All’].
title (str, optional) – The title for the plot. Defaults to “Global Genes”.
mask_path (str, optional) – Path to the mask image file. If specified, only genes within the mask will be plotted. Defaults to None.
improved (bool, optional) – Whether to use improved puncta with gene data or retrieve all puncta. Defaults to False.
save (bool, optional) – Whether to save the plot to a file instead of displaying it. If True, the plot will be saved as a .png file in the ‘inspect_gene’ subdirectory of args.puncta_path. Defaults to False.
- Returns:
This function doesn’t return any value. It either displays the plot or saves it to a .png file depending on the ‘save’ parameter.
- exm.puncta.inspect.plot_genes_global_zstep(args, tileset, gene_list=['All'], title='Global Genes z-step', improved=False)[source]
Generates an animated .mp4 video depicting the global distribution of specified genes in a 3D volume using z-step slices. If ‘All’ is specified in gene_list, all possible genes will be included.
- Parameters:
args (args.Args instance) – Configuration options.
tileset (exm.stitching.tileset.Tileset) – The tileset object from which slices will be displayed.
gene_list (list) – List of genes to be plotted. If [‘All’] is passed, all possible genes will be plotted. Defaults to [‘All’].
title (str, optional) – The title for the animation. Defaults to “Global Genes z-step”.
improved (bool, optional) – Whether to use improved puncta with gene data or retrieve all puncta. Defaults to False.
- Returns:
This function doesn’t return any value. It creates and saves an animation of gene distributions over z-slices to the ‘inspect_gene’ subdirectory of args.puncta_path as a .mp4 file.
Puncta Improve Detection
The Puncta Detection Improvement Module is geared towards refining the accuracy of detected puncta in fluorescence microscopy images. It achieves this by utilizing distance-based criteria to consolidate puncta across different channels and enhancing the detection of local maxima, which are indicative of puncta locations.
- exm.puncta.improve.find_nearest_points(point_cloud1, point_cloud2, distance_threshold=14)[source]
Finds the nearest points between two point clouds based on Euclidean distance. This function computes the Euclidean distance between each point in point_cloud1 and point_cloud2. It then finds the nearest point in point_cloud2 for each point in point_cloud1, within the specified distance_threshold.
- Parameters:
point_cloud1 (np.ndarray) – An Nx3 numpy array representing the first point cloud. Each row is a point, and the columns represent the x, y, and z coordinates respectively.
point_cloud2 (np.ndarray) – An Mx3 numpy array representing the second point cloud. Each row is a point, and the columns represent the x, y, and z coordinates respectively.
distance_threshold (float, optional) – The maximum allowed distance for points to be considered as ‘nearest’. Points in point_cloud2 that are farther away from a point in point_cloud1 than this threshold will not be considered as ‘nearest’. Default is 14.
- Returns:
A list of dictionaries. Each dictionary represents a pair of nearest points and contains two keys: ‘point0’ and ‘point1’. ‘point0’ is the index of a point in point_cloud1, and ‘point1’ is the index of its nearest point in point_cloud2 within the distance_threshold.
- exm.puncta.improve.find_nearest_puncta(args, position, fov, code, center_dist=15, distance_threshold=8, GPU=False)[source]
Performs distance-thresholded puncta consolidation across channels. Optionally, it can utilize GPU acceleration.
- Parameters:
args – Configuration options.
position (tuple) – (d0, d1, d2) coordinates of the position.
fov (int) – Field of view.
code (int) – The code of the volume chunk to be searched.
center_dist (int, optional) – Size of the ROI around the input position in the d1 and d2 directions. Defaults to 15.
distance_threshold (int, optional) – Maximum allowed Euclidean distance from the input position to a maximum point. Defaults to 8.
GPU (bool, optional) – Whether to use GPU acceleration. Defaults to False.
- Returns:
A dictionary containing: - ‘c0’ to ‘c3’: Map to another dictionary containing ‘position’, ‘intensity’, and ‘distance’ of the closest maximum point in this channel. - ‘intensity’: A list of intensities for all channels. - ‘color’: The channel with the maximum intensity. - ‘position’: The position of the maximum point in the ‘color’ channel.
If no maximum points were found in any channel, an empty dictionary is returned.
- exm.puncta.improve.improve_nearest(args, fov, num_missing_code=4)[source]
This function retrieves all puncta within a specified field of view (fov) and refines each puncta’s information by adding the nearest puncta for all missing codes in the puncta’s barcode. The refined puncta information is then saved to a pickle file in a subdirectory corresponding to the given fov.
- Parameters:
- Returns:
This function doesn’t return any value. However, it does save the improved puncta list to a pickle file named ‘nearest_improved_puncta.pickle’ in a subdirectory named ‘fov{fov}’ under the directory specified by args.puncta_path.
- exm.puncta.improve.puncta_all_nearest_points(args, puncta)[source]
This function finds the nearest puncta for all missing codes in a given puncta’s barcode by iteratively calling the ‘find_nearest_puncta’ function for each missing code.
- Parameters:
args (args.Args instance) – Configuration options.
puncta (dict) – A dictionary containing information about a puncta, including its ‘barcode’, ‘position’, and ‘fov’ values after consolidate codes.
- Returns:
A dictionary where keys are the names of the missing codes (‘code0’ to ‘codeN’). Each key maps to another dictionary containing ‘position’ (the position of the nearest puncta for the corresponding code), ‘intensity’ (the intensity of the nearest puncta), ‘distance’ (the distance to the nearest puncta), and ‘ref_code’ (the reference code of the nearest puncta). If no nearest puncta were found for any missing code, an empty dictionary is returned.
- exm.puncta.improve.puncta_nearest_points(args, fov, puncta_index, search_code, verbose=True)[source]
Identifies and retrieves the nearest puncta points based on the provided puncta index and search code. The function finds the reference code, generates two point clouds (one for the original puncta and another for potential new puncta points), and identifies the closest pairs between these two point clouds. This process is repeated to find the closest points to the newly identified puncta points.
- Parameters:
args – Configuration options.
fov (int) – The field of view (fov) to consider.
puncta_index (int) – The index of the puncta to start the search from.
search_code (int) – The search code to use when finding new puncta points.
mute (bool, optional) – If set to True, the function will not print progress messages. If set to False, the function will print progress messages.
- Returns:
A tuple containing the reference code, the closest new puncta point, and the closest point to the new puncta point.
Puncta Base-Calling
The Puncta Base-calling Module is a post-processing tool for fluorescence expansion microscopy data, designed to associate detected puncta with specific genes and nuclei based on predefined criteria. This enhances the biological interpretation of the imaging data by connecting puncta, which represent molecular targets, to genetic information and cellular structures.
- exm.puncta.basecalling.puncta_assign_gene(args, fov, option='original')[source]
Assign genes to detected puncta based on hamming distance from barcodes.
This function maps genes to the detected puncta for a given field of view (fov) by comparing the hamming distance from predefined barcodes. It retrieves the barcode mappings for genes, and for each puncta in the given fov, assigns a gene based on the closest hamming distance.
- Parameters:
- Returns:
None. Results are saved as a pickle file.
- Return type:
None
- exm.puncta.basecalling.puncta_assign_nuclei(args, fov, distance_threshold=100, compare_to_nuclei_surface=True, num_nearest_nuclei=3, option='original')[source]
Assign puncta to nuclei based on their spatial proximity. For each puncta, the function determines the corresponding nucleus (if any) based on a set distance threshold.
- Parameters:
args (Args) – Configuration options. This should be an instance of the Args class.
fov (int) – Field of view.
distance_threshold (float, default is 100) – Threshold for the maximum allowable distance between puncta and a nucleus.
compare_to_nuclei_surface (bool, default is True) – Flag to determine if distance should be computed to the nuclei surface.
num_nearest_nuclei (int, default is 3) – Number of nearest nuclei to consider when comparing to the nuclei surface.
option (str, default is 'original') – Option to determine which puncta data to use (‘original’ or ‘improved’).
- Raises:
Logs an error message if there’s an issue during processing.
- Return type:
None
Puncta Benchmark
- exm.puncta.benchmark.aggregate_puncta_analysis(args, improved=False)[source]
Aggregates puncta analysis results across all fields of view (fovs). This function opens pickle files containing the results of puncta analysis for each fov specified in the args parameter. It then aggregates these results, summing or unionizing values as appropriate based on the data type. The aggregated results are written to a pickle file with a specific name based on whether the original puncta data was improved or not.
- Parameters:
args (args.Args) – Configuration options. The fovs attribute should contain a list of fovs to include in the aggregation.
improved (bool) – If True, the function retrieves data from ‘improved_puncta_analysis.pickle’ files, else retrieves data from ‘original_puncta_analysis.pickle’ files. Default is False.
- Returns:
No return value. The function saves the aggregated analysis result to a pickle file named ‘aggregate_improved_puncta_analysis.pickle’ or ‘aggregate_orginal_puncta_analysis.pickle’ in the directory specified by args.puncta_path, depending on the improved parameter.
- exm.puncta.benchmark.plot_benchmark_data(args, mode='single', fov=None)[source]
Generates a comparative dashboard for puncta analysis results in a single field of view (fov) or across all fovs. This function opens pickle files containing the original and improved results of puncta analysis for a specified fov or for all fovs. The results are used to generate various bar charts showing comparisons between original and improved puncta counts, distributions, and other metrics. These charts are saved as a HTML file.
- Parameters:
- Returns:
No return value. The function saves the generated dashboard to a HTML file named ‘puncta_comparison_dashboard.html’ or ‘overall_puncta_comparison_dashboard.html’ in the directory specified by args.puncta_path, depending on the mode parameter.
- exm.puncta.benchmark.puncta_analysis(args, fov, improved=False)[source]
Analyzes puncta within a specified field of view (fov). This function opens a pickle file containing consolidated puncta data, and performs an analysis to collect statistics on these puncta. The statistics include total puncta, puncta with and without reference codes, reference code counts, puncta with and without missing codes, missing code counts, unique reference codes, puncta without missing codes and gene, etc. The results are stored in a dictionary and written to a pickle file with a specific file name based on whether the puncta data was improved or not.
- Parameters:
- Returns:
No return value. The function saves the analysis result to a pickle file named ‘improved_puncta_analysis.pickle’ or ‘original_puncta_analysis.pickle’ in a subdirectory of each fov, depending on the improved parameter.