{ "cells": [ { "cell_type": "markdown", "id": "d40d2fc4", "metadata": {}, "source": [ "\n", "# PDRs4All: Creating NIRCam Imaging Data\n", "---\n", "**Author**: Amélie Canin (amelie.canin@irap.omp.eu) \n", "\n", "**Latest Update**: 15 juin 2022\n", "\n", "This notebook follows MIRAGE example from STScI's available on the [Git](https://github.com/spacetelescope/mirage) and used some documentations (see [documentation](#doc) to find useful links)." ] }, { "cell_type": "markdown", "id": "d52b4b83", "metadata": {}, "source": [ "
\n", "

Notebook Goals

\n", " \n", " \n", "
" ] }, { "cell_type": "markdown", "id": "cb8636d7", "metadata": {}, "source": [ "## Introduction" ] }, { "cell_type": "markdown", "id": "70b30c15", "metadata": {}, "source": [ "\n", "\n", "### The PDRs4All: Radiative feeback from massive stars\n", "\n", "PDRs4All is a program of imaging and spectroscopy in the Orion Bar to understand the impact of massive stars. \n", "Massive stars disrupt their natal molecular cloud material by dissociating molecules, ionizing atoms and molecules, and heating the gas and dust. These processes drive the evolution of interstellar matter in our Galaxy and throughout the Universe from the era of vig- orous star formation at redshifts of 1-3, to the present day. Much of this interaction occurs in Photo-Dissociation Regions where far-ultraviolet photons of these stars create a largely neutral, but warm region of gas and dust.\n", "\n", "PDRs4All has 12 NIRCam imaging filters in the same field of view to observe different Polycyclic Aromatic Hydrocarbons on the Orion Bar in the field of view on the [figure below](#fov). More detailled of the PDRs4All are available in the [Berné, Habart, Peeters et al. 2022](https://iopscience.iop.org/article/10.1088/1538-3873/ac604c) and on the [website](https://pdrs4all.org)." ] }, { "cell_type": "markdown", "id": "d45fa4f1", "metadata": {}, "source": [ "\n", "" ] }, { "cell_type": "markdown", "id": "5cf5caf0", "metadata": {}, "source": [ "In this notebook, we will simulate one NIRCam observation for the filter F140M and demonstrate the whole processus to create raw images using MIRAGE (Multi-Instrument RAmp GEnerator)." ] }, { "cell_type": "markdown", "id": "3cae2a67", "metadata": {}, "source": [ "\n", "## Resources and Documentation" ] }, { "cell_type": "markdown", "id": "4716ddeb", "metadata": {}, "source": [ "There are several places to find information on intalling and running MIRAGE. This notebook will provides examples of running MIRAGE with a background image, but will not demonstrate all options and features. Please see the following links for more in-depth documentations.\n", "\n", "* [JWST Documentation (JDox) for MIRAGE](https://jwst-docs.stsci.edu/jwst-other-tools/mirage-jwst-data-simulator)\n", "\n", "* [MIRAGE Documentation](https://mirage-data-simulator.readthedocs.io/en/latest/)\n", "\n", "* [`mirage` package GitHub repository](https://github.com/spacetelescope/mirage)\n", "\n", "* [Help Desk](https://stsci.service-now.com/jwst?id=sc_category&sys_id=e15706fc0a0a0aa7007fc21e1ab70c2f): If you have any questions or problems regarding MIRAGE or others, submit a ticket to the Help Desk" ] }, { "cell_type": "markdown", "id": "7e757d5f", "metadata": {}, "source": [ "### Installation" ] }, { "cell_type": "markdown", "id": "9c967000", "metadata": {}, "source": [ "
\n", " Before running this notebook, you will have to first install the mirage package.\n", " \n", "**NOTE:** The `mirage` package requries Python 3.8+

\n", " \n", "The recommended way to install the pipeline is via `pip`. Follow the steps below to create a new conda environment, activate that environment, and then install the latest released version of the mirage. You can name your environment anything you like. In the lines below, replace `` with your chosen environment name.\n", "\n", ">`conda create -n python`
\n", ">`conda activate `
\n", ">`pip install mirage`\n", "\n", "For more detailed instructions on the various ways to install the package, including installing more recent development versions of mirage, see the [installation instructions](https://mirage-data-simulator.readthedocs.io/en/latest/install.html?highlight=git) on GitHub.\n", " \n", "
" ] }, { "cell_type": "markdown", "id": "ce7fe2a0", "metadata": {}, "source": [ "### Reference Files" ] }, { "cell_type": "markdown", "id": "b3282043", "metadata": {}, "source": [ "Reference files are used to simulated the NIRCam images, to create the dark current etc. The same reference files used in the Calibration Pipeline. Others files are specific to MIRAGE.\n", "\n", "When running MIRAGE, it will automatically look for any required reference files in a pre-defined local directory. If the required reference files are not present, they will automatically be downloaded from the Calibration Reference Data System (CRDS) at STScI.\n", " \n", "
\n", " \n", "You will have to specify a local directory in which to store reference files, along with the server to use to download the reference files from CRDS. To accomplish this, there are three environment variables that should be set prior to calling MIRAGE. These are the `CRDS_PATH`, `CRDS_SERVER_URL` and `MIRAGE_DATA` variables. In the example below, reference files will be downloaded to the \"crds_cache\" directory under the home directory.\n", "\n", ">`$ export CRDS_PATH=$HOME/crds_cache`
\n", ">`$ export CRDS_SERVER_URL=https://jwst-crds-pub.stsci.edu`
\n", ">`$ export MIRAGE_DATA=$HOME/mirage_data`
\n", "OR:
\n", "`os.environ[\"CRDS_PATH\"] = \"/user/myself/crds_cache\"`
\n", "`os.environ[\"CRDS_SERVER_URL\"] = \"https://jwst-crds-pub.stsci.edu\"`
\n", "`os.environ[\"MIARGE_DATA\"] = \"/user/myself/mirage_data\"`
\n", "\n", "The first time you run MIRAGE, the CRDS server should download all of the context and reference files that are needed. \n", "
\n", "\n", "Note:The MIRAGE_DATA directory requires a lot of space in your disk." ] }, { "cell_type": "code", "execution_count": null, "id": "7fcfe3f9", "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "# Make sure to replace with the path to your CRDS cache and mirage data directories\n", "os.environ[\"MIRAGE_DATA\"] = \"/path/to/mirage/data/\"\n", "os.environ[\"CRDS_SERVER_URL\"] = \"https://jwst-crds-pub.stsci.edu\"\n", "os.environ[\"CRDS_PATH\"] = \"/path/to/crds/files/\"\n", "os.environ[\"CRDS_CONTEXT\"] = \"jwst_0862.pmap\"\n" ] }, { "cell_type": "markdown", "id": "bcc65738", "metadata": {}, "source": [ "The following cell may or may not be needed depending on the latest version of MIRAGE." ] }, { "cell_type": "code", "execution_count": null, "id": "372efe5c-c0ce-41a4-b650-e13b5276b8ef", "metadata": {}, "outputs": [], "source": [ "# Installing additional modules\n", "\n", "import sys\n", "!{sys.executable} -m pip install jwst-gtvt -q\n", "import jwst_gtvt" ] }, { "cell_type": "markdown", "id": "ecca4609", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": null, "id": "7aa3416e", "metadata": {}, "outputs": [], "source": [ "import yaml\n", "\n", "# Steps mirage\n", "from mirage.dark.dark_prep import DarkPrep\n", "from mirage.ramp_generator.obs_generator import Observation\n", "from mirage.seed_image.fits_seed_image import ImgSeed\n", "from mirage.yaml import yaml_generator\n", "\n", "import jwst\n", "from astropy.io import fits\n" ] }, { "cell_type": "markdown", "id": "be4f9a15", "metadata": {}, "source": [ "Set up matplotlib for plotting" ] }, { "cell_type": "code", "execution_count": null, "id": "a6e2646c", "metadata": {}, "outputs": [], "source": [ "import matplotlib.pyplot as plt\n", "from matplotlib import rcParams\n", "\n", "# Use this version for non-interactive plots (easier scrolling of the notebook)\n", "#%matplotlib inline\n", "\n", "# Use this version if you want interactive plots\n", "#%matplotlib notebook\n", "%matplotlib inline\n", "\n", "# These gymnastics are needed to make the sizes of the figures\n", "# be the same in both the inline and notebook versions\n", "#%config InlineBackend.print_figure_kwargs = {'bbox_inches': None}\n", "\n", "# You may want to change the following configurations to customize \n", "# figure sizes and resolutions\n", "rcParams['figure.figsize'] = [11,8]\n", "rcParams['figure.dpi'] = 80\n", "rcParams['savefig.dpi'] = 80" ] }, { "cell_type": "markdown", "id": "d716705c", "metadata": {}, "source": [ "Check which version of mirage you are running" ] }, { "cell_type": "code", "execution_count": null, "id": "38cd4ca8", "metadata": {}, "outputs": [], "source": [ "import mirage\n", "print(mirage.__version__)" ] }, { "cell_type": "markdown", "id": "c28fb411", "metadata": {}, "source": [ "Set the directories for outputs" ] }, { "cell_type": "code", "execution_count": null, "id": "9a0b4c32", "metadata": {}, "outputs": [], "source": [ "output_dir = './output2/f140m'\n", "simdata_dir = os.path.join(output_dir,'stage0')\n", "\n", "if not os.path.exists(output_dir):\n", " os.makedirs(output_dir)" ] }, { "cell_type": "markdown", "id": "2c289ffb", "metadata": {}, "source": [ "Set the path of the image used in background. You can download it here: \n", "[Orion_bar.fits](https://stsci.box.com/s/k0ilbd562rft77v9lq2to4b1qmt9fm3h)" ] }, { "cell_type": "code", "execution_count": null, "id": "92c0fb94", "metadata": {}, "outputs": [], "source": [ "mosaic_file = './Orion_bar.fits'" ] }, { "cell_type": "markdown", "id": "f64ff918", "metadata": {}, "source": [ "Set files containing observation information as filters used, number of groups, dithers, etc.\n", "The xml and pointing files can be downloaded here: \n", "[ERS1288_nircam.xml](https://stsci.box.com/s/1si8otf9rtqnyvzbz3dajveyayz5193m) \n", "[ERS1288_nircam.pointing](https://stsci.box.com/s/keb9hvimsrmxzud67qjbry2d7hghejsx)\n", "\n", "
\n", "\n", "The `xml` and `pointing` files can be found in APT.\n", " \n", "To do this, in APT, when the observation is open, go to `File/Export` and then select `xml`. To export `pointing` you have to do the same one more time selecting `pointing`.\n", "
\n" ] }, { "cell_type": "code", "execution_count": null, "id": "74704559", "metadata": {}, "outputs": [], "source": [ "xml_file = './ERS1288_nircam.xml'\n", "pointing_file = xml_file.replace('.xml', '.pointing')" ] }, { "cell_type": "markdown", "id": "36749c45", "metadata": {}, "source": [ "Function to show the different images" ] }, { "cell_type": "code", "execution_count": null, "id": "87bdf298", "metadata": {}, "outputs": [], "source": [ "def show(array,title,min=0,max=1000):\n", " plt.figure(figsize=(10,10))\n", " plt.imshow(array,clim=(min,max),origin='lower')\n", " plt.title(title)\n", " plt.colorbar()" ] }, { "cell_type": "markdown", "id": "7f75e25b", "metadata": {}, "source": [ "## Mosaic image" ] }, { "cell_type": "markdown", "id": "6b17ef84", "metadata": {}, "source": [ "We use a mosaic image as background for the short wavelength: the Hubble Space Telescope image of the Orion Nebula at 1.4 μm obtained by [Robberto et al 2020](https://iopscience.iop.org/article/10.3847/1538-4357/ab911e) with the Wide Field Camera 3 (WFC3)." ] }, { "cell_type": "markdown", "id": "0c760ec1", "metadata": {}, "source": [ "Store the mosaic image in the variable data" ] }, { "cell_type": "code", "execution_count": null, "id": "2eb3c428", "metadata": {}, "outputs": [], "source": [ "data = fits.getdata(mosaic_file)" ] }, { "cell_type": "markdown", "id": "00a37501", "metadata": {}, "source": [ "Plot the image" ] }, { "cell_type": "code", "execution_count": null, "id": "cd324ae8", "metadata": {}, "outputs": [], "source": [ "show(data,'HST Mosaic',max=200)" ] }, { "cell_type": "markdown", "id": "e942cf56", "metadata": {}, "source": [ "## YAML generator" ] }, { "cell_type": "markdown", "id": "42bca6c9", "metadata": {}, "source": [ "**Description**\n", "\n", "The first stage of the MIRAGE simulation is the creation of YAML files from the inputs above, especially the files exported from APT. YAML is a language used for configuration files as XML or JSON. The YAML files contain all the information needed for MIRAGE to work properly and simulate the right observations like the telescope and instrument set- tings, the reference files. The YAML generator allows to execute this first step. The YAML generator takes as inputs the xml and pointing files created with APT, the roll angle, the datatype set as raw to obtain the same results as the JWST, and the output directories.\n", "\n", "**Inputs**\n", "\n", "* Files from APT (`.xml` and `.pointing`) containing the observation information\n", "* Output directory where to store the yaml files\n", "* Simulation data output directory (`simdata_output_dir`) where to store the MIRAGE simulation\n", "* Roll angle corresponding to the telescope V3 angle\n", "\n", "**Outputs**\n", "\n", "* YAML files (`.yaml`) resulting from yaml generation one for each detector, each filter and each dither\n" ] }, { "cell_type": "markdown", "id": "8609266c", "metadata": {}, "source": [ "**Roll angle**\n", "\n", "The roll angle of the telescope is needed so that MIRAGE can position the field of view correctly on the sky. It corresponds to the position angle of the V3 axis in degrees east of north." ] }, { "cell_type": "code", "execution_count": null, "id": "4257eab3", "metadata": {}, "outputs": [], "source": [ "roll_angle = yaml_generator.all_obs_v3pa_on_date(pointing_file, date='2022-09-10')\n", "print(roll_angle)" ] }, { "cell_type": "markdown", "id": "e2f66f96", "metadata": {}, "source": [ "**YAML generator**" ] }, { "cell_type": "code", "execution_count": null, "id": "efa6ae41", "metadata": {}, "outputs": [], "source": [ "yam = yaml_generator.SimInput(xml_file, pointing_file, verbose=True,\n", " output_dir=output_dir,\n", " simdata_output_dir=simdata_dir,\n", " roll_angle=roll_angle,\n", " datatype='raw'\n", " )\n", "yam.use_linearized_darks = True\n", "yam.create_inputs()" ] }, { "cell_type": "markdown", "id": "87546302", "metadata": {}, "source": [ "We chose one observation." ] }, { "cell_type": "code", "execution_count": null, "id": "dba9d920", "metadata": {}, "outputs": [], "source": [ "yamlfile = './output2/f140m/jw01288001001_01101_00021_nrcb1.yaml'" ] }, { "cell_type": "markdown", "id": "9cbcd5c8", "metadata": {}, "source": [ "Open of the yaml into a dictionnaire" ] }, { "cell_type": "code", "execution_count": null, "id": "7f935573", "metadata": {}, "outputs": [], "source": [ "with open(yamlfile) as f:\n", " yaml_dict = yaml.safe_load(f)\n", "yaml_dict" ] }, { "cell_type": "markdown", "id": "f256e33b", "metadata": {}, "source": [ "Find the filter of the observation" ] }, { "cell_type": "code", "execution_count": null, "id": "ad30ef65", "metadata": {}, "outputs": [], "source": [ "yaml_dict['Readout']['filter']" ] }, { "cell_type": "markdown", "id": "5f0ce2f5", "metadata": {}, "source": [ "## Simulation" ] }, { "cell_type": "markdown", "id": "3dd3c384", "metadata": {}, "source": [ "**Description**\n", "\n", "The JWST Multi Instrument Ramp Generator (MIRAGE), is a simulator to create JWST-like NIRCam images. It is an open source Python package developed by the STScI [Hilbert et al. 2019](https://zenodo.org/record/3519262#.Yqss8C0itUc).\n", "\n", "This simulator is divided in three stages: creation of the seed image; creation of the dark current; creation of the final observation. See the figures on this [page](https://mirage-data-simulator.readthedocs.io/en/latest/three_steps.html).\n", "\n", "The simulation needs an YAML file corresponding to the observation, containing the filter, the detector, the roll angle among others. This simulator can be used with a stellar catalog or an image used in background.\n", "\n", "In this notebook, we will only present the method using an image as background.\n", "\n", "\n", "**Inputs**\n", "\n", "* YAML file (`.yaml`) resulting from yaml generation\n", "\n", "**Outputs**\n", "\n", "* A raw exposure (`*uncal.fits`) containing the 4-dimensional raw data from all detector readouts: (nintegrations x ngroups x ncols x nrows)\n", "\n" ] }, { "cell_type": "markdown", "id": "ee912fb3", "metadata": {}, "source": [ "### Seed image" ] }, { "cell_type": "markdown", "id": "5d54857d", "metadata": {}, "source": [ "The seed image corresponds to an idealized, noiseless image of the scene. As our extended images are bigger than the detectors, MIRAGE crops the scene.\n", "\n", "**Inputs**\n", "\n", "* YAML file (`.yaml`) resulting from yaml generation\n", "* An image used in background \n", "\n", "**Outputs**\n", "\n", "* A seed image (`*botted_seed_image.fits`)" ] }, { "cell_type": "code", "execution_count": null, "id": "ff564f5a", "metadata": {}, "outputs": [], "source": [ "# Define output filenames and directories\n", "sim_data_dir = yaml_dict['Output']['directory']\n", "simulated_filename = yaml_dict['Output']['file']\n", "crop_file = simulated_filename.replace('.fits', '_cropped_from_mosaic.fits')\n", "crop_file = os.path.join(sim_data_dir, crop_file)\n", "blot_file = simulated_filename.replace('.fits', '_blotted_seed_image.fits')" ] }, { "cell_type": "code", "execution_count": null, "id": "a4516f76", "metadata": {}, "outputs": [], "source": [ "# Instantiate the seed image\n", "seed = ImgSeed(paramfile=yamlfile, mosaic_file=mosaic_file, \n", " cropped_file=crop_file,\n", " outdir=sim_data_dir, blotted_file=blot_file, mosaic_fwhm=0.009,\n", " mosaic_fwhm_units='arcsec', gaussian_psf=True)" ] }, { "cell_type": "code", "execution_count": null, "id": "c74e1453", "metadata": {}, "outputs": [], "source": [ "# Creation of the seed image\n", "seed.crop_and_blot()" ] }, { "cell_type": "code", "execution_count": null, "id": "6eaa9e7a", "metadata": {}, "outputs": [], "source": [ "# Shape of the seed image\n", "seed.seed_image.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "880a8314", "metadata": {}, "outputs": [], "source": [ "# Plot the seed image\n", "show(seed.seed_image,'Seed image',max=200)" ] }, { "cell_type": "markdown", "id": "69bb5364", "metadata": {}, "source": [ "### Dark current" ] }, { "cell_type": "markdown", "id": "064dd862", "metadata": {}, "source": [ "This stage prepares the dark current exposure for the observation.\n", "\n", "**Inputs**\n", "\n", "* YAML file (`.yaml`) resulting from yaml generation\n", "\n", "**Outputs**\n", "\n", "* A dark current (`*dark_prep_object.fits`)" ] }, { "cell_type": "code", "execution_count": null, "id": "384b8ad1", "metadata": {}, "outputs": [], "source": [ "dark = DarkPrep()" ] }, { "cell_type": "code", "execution_count": null, "id": "24968098", "metadata": {}, "outputs": [], "source": [ "dark.paramfile = yamlfile\n", "dark.prepare()" ] }, { "cell_type": "markdown", "id": "62fcf6e7", "metadata": {}, "source": [ "### Observation" ] }, { "cell_type": "markdown", "id": "0eaa19c4", "metadata": {}, "source": [ "This stage produces the final raw image with the noise from the background but also due to the detectors. This stage combines the seed im- age and the dark current exposure.\n", "\n", "**Inputs**\n", "\n", "* YAML file (`.yaml`) resulting from yaml generation\n", "* A seed image (`*botted_seed_image.fits`)\n", "* A dark current (`*dark_prep_object.fits`)\n", "\n", "**Outputs**\n", "\n", "* A raw exposure (`*uncal.fits`) containing the 4-dimensional raw data from all detector readouts: (nintegrations x ngroups x ncols x nrows)." ] }, { "cell_type": "code", "execution_count": null, "id": "667771a2", "metadata": {}, "outputs": [], "source": [ "# Instantiate the observation\n", "obs = Observation()" ] }, { "cell_type": "code", "execution_count": null, "id": "2ef56ba7", "metadata": {}, "outputs": [], "source": [ "# Define parameters of the observation\n", "obs.paramfile = yamlfile \n", "obs.seed = seed.seed_image\n", "obs.segmap = seed.seed_segmap\n", "obs.seedheader = seed.seedinfo\n", "obs.linDark = dark.prepDark" ] }, { "cell_type": "code", "execution_count": null, "id": "9ead25f0", "metadata": {}, "outputs": [], "source": [ "# Creation of the observation\n", "obs.create()" ] }, { "cell_type": "code", "execution_count": null, "id": "9e01b73b", "metadata": {}, "outputs": [], "source": [ "# Path of the observation\n", "obs.raw_output" ] }, { "cell_type": "code", "execution_count": null, "id": "aeada87a", "metadata": {}, "outputs": [], "source": [ "# Shape of the observation: (nintegrations x ngroups x ncols x nrows)\n", "obs.raw_outramp.shape" ] }, { "cell_type": "code", "execution_count": null, "id": "ab81b4a9", "metadata": {}, "outputs": [], "source": [ "# Plot of one group of the observation\n", "show(obs.raw_outramp[0,0,:,:],'Observation',max=20000)" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3 (ipykernel)", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.13" } }, "nbformat": 4, "nbformat_minor": 5 }