arrakis.utils.pipeline ====================== .. py:module:: arrakis.utils.pipeline .. autoapi-nested-parse:: Pipeline and flow utility functions .. !! processed by numpydoc !! Attributes ---------- .. autoapisummary:: arrakis.utils.pipeline.SUPPORTED_IMAGE_TYPES arrakis.utils.pipeline.TQDM_OUT arrakis.utils.pipeline.logo_str Classes ------- .. autoapisummary:: arrakis.utils.pipeline.TqdmProgressBar arrakis.utils.pipeline.performance_report_prefect Functions --------- .. autoapisummary:: arrakis.utils.pipeline.chunk_dask arrakis.utils.pipeline.cpu_to_use arrakis.utils.pipeline.delayed_to_da arrakis.utils.pipeline.generic_parser arrakis.utils.pipeline.inspect_client arrakis.utils.pipeline.port_forward arrakis.utils.pipeline.submit_task_with_rate_limit arrakis.utils.pipeline.tqdm_dask arrakis.utils.pipeline.upload_image_as_artifact_task arrakis.utils.pipeline.workdir_arg_parser Module Contents --------------- .. py:class:: TqdmProgressBar(keys, scheduler=None, interval='100ms', loop=None, complete=True, start=True, **tqdm_kwargs) Bases: :py:obj:`distributed.diagnostics.progressbar.ProgressBar` Tqdm for Dask .. !! processed by numpydoc !! .. py:method:: _draw_bar(remaining, all, **kwargs) .. py:method:: _draw_stop(**kwargs) .. py:attribute:: loop .. py:attribute:: tqdm .. py:class:: performance_report_prefect(filename='dask-report.html', stacklevel=1, mode=None, storage_options=None) Gather performance report from prefect_dask Basically stolen from: https://distributed.dask.org/en/latest/_modules/distributed/client.html#performance_report This creates a static HTML file that includes many of the same plots of the dashboard for later viewing. The resulting file uses JavaScript, and so must be viewed with a web browser. Locally we recommend using ``python -m http.server`` or hosting the file live online. :Parameters: * **filename** (*str, optional*) -- The filename to save the performance report locally * **stacklevel** (*int, optional*) -- The code execution frame utilized for populating the Calling Code section of the report. Defaults to `1` which is the frame calling ``performance_report_prefect`` * **mode** (*str, optional*) -- Mode parameter to pass to :func:`bokeh.io.output.output_file`. Defaults to ``None``. * **storage_options** (*dict, optional*) -- Any additional arguments to :func:`fsspec.open` when writing to a URL. .. rubric:: Examples >>> with performance_report_prefect(filename="myfile.html", stacklevel=1): ... x.compute() .. !! processed by numpydoc !! .. py:method:: __aenter__() :async: .. py:method:: __aexit__(exc_type, exc_value, traceback, code=None) :async: .. py:method:: __enter__() .. py:method:: __exit__(exc_type, exc_value, traceback) .. py:attribute:: _stacklevel :value: 1 .. py:attribute:: filename :value: 'dask-report.html' .. py:attribute:: mode :value: None .. py:attribute:: storage_options .. py:function:: chunk_dask(outputs: list, batch_size: int = 10000, task_name='', progress_text='', verbose=True) -> list .. py:function:: cpu_to_use(max_cpu: int, count: int) -> int Find number of cpus to use. Find the right number of cpus to use when dividing up a task, such that there are no remainders. :Parameters: * **max_cpu** (*int*) -- Maximum number of cores to use for a process. * **count** (*int*) -- Number of tasks. :returns: Maximum number of cores to be used that divides into the number .. !! processed by numpydoc !! .. py:function:: delayed_to_da(list_of_delayed: list[dask.delayed.Delayed], chunk: int | None = None) -> dask.array.Array Convert list of delayed arrays to a dask array :Parameters: * **list_of_delayed** (*List[delayed]*) -- List of delayed objects * **chunk** (*int, optional*) -- Chunksize to use. Defaults to None. :returns: Dask array :rtype: da.Array .. !! processed by numpydoc !! .. py:function:: generic_parser(parent_parser: bool = False) -> argparse.ArgumentParser .. py:function:: inspect_client(client: dask.distributed.Client | None = None) -> tuple[str, int, int, astropy.units.Quantity, int, astropy.units.Quantity] _summary_ :Parameters: **client** (*Union[distributed.Client,None]*) -- Dask client to inspect. if None, will use the default client. :returns: addr, nworkers, nthreads, memory, threads_per_worker, memory_per_worker :rtype: Tuple[ str, int, int, u.Quantity, float, u.Quantity ] .. !! processed by numpydoc !! .. py:function:: port_forward(port: int, target: str) -> None Forward ports to local host :Parameters: * **port** (*int*) -- port to forward * **target** (*str*) -- Target host .. !! processed by numpydoc !! .. py:function:: submit_task_with_rate_limit(task: prefect.Task, *args, **kwargs) -> prefect.futures.PrefectFuture Submit a task with rate limiting :Parameters: * **task** (*Task*) -- Task to submit * **\*args** -- Arguments to pass to the task * **\*\*kwargs** -- Keyword arguments to pass to the task :returns: Future object :rtype: PrefectFuture .. !! processed by numpydoc !! .. py:function:: tqdm_dask(futures_in: dask.distributed.Future, **kwargs) -> None Tqdm for Dask futures .. !! processed by numpydoc !! .. py:function:: upload_image_as_artifact_task(image_path: pathlib.Path, description: str | None = None) -> uuid.UUID Create and submit a markdown artifact tracked by prefect for an input image. Currently supporting png formatted images. The input image is converted to a base64 encoding, and embedded directly within the markdown string. Therefore, be mindful of the image size as this is tracked in the postgres database. :Parameters: * **image_path** (*Path*) -- Path to the image to upload * **description** (*Optional[str], optional*) -- A description passed to the markdown artifact. Defaults to None. :returns: Generated UUID of the registered artifact :rtype: UUID .. !! processed by numpydoc !! .. py:function:: workdir_arg_parser(parent_parser: bool = False) -> argparse.ArgumentParser .. py:data:: SUPPORTED_IMAGE_TYPES :value: ('png',) .. py:data:: TQDM_OUT .. py:data:: logo_str :value: Multiline-String .. raw:: html
Show Value .. code-block:: python """ mmm mmm mmm mmm mmm )-( )-( )-( )-( )-( ( S ) ( P ) ( I ) ( C ) ( E ) | | | | | | | | | | |___| |___| |___| |___| |___| mmm mmm mmm mmm )-( )-( )-( )-( ( R ) ( A ) ( C ) ( S ) | | | | | | | | |___| |___| |___| |___| """ .. raw:: html