{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Valadating the SPIRE red catalogue\n",
    "\n",
    "![HELP LOGO](https://avatars1.githubusercontent.com/u/7880370?s=100&v=4>)\n",
    "\n",
    "### three methods of selecting 500um riser galaxies\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "%matplotlib inline\n",
    "from astropy.table import Table, vstack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "all_names =  ['GAMA-09','GAMA-12','GAMA-15','HATLAS-NGP','HATLAS-SGP','SSDF',\\\n",
    "              'AKARI-SEP','Bootes','CDFS-SWIRE','COSMOS','EGS',\\\n",
    "              'ELAIS-N1','ELAIS-N2','ELAIS-S1','HDF-N','Lockman-SWIRE','SA13',\\\n",
    "              'SPIRE-NEP','xFLS','XMM-13hr','XMM-LSS','AKARI-NEP']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Combine all red catalogues\n",
    "name = 'data/'+all_names[0]+'_SPIRE_D_MF_cat.fits'\n",
    "cat=Table.read(name)\n",
    "for j in range(1,np.size(all_names)):\n",
    "    name = 'data/'+all_names[j]+'_SPIRE_D_MF_cat.fits'\n",
    "    cat2=Table.read(name)\n",
    "    cat = vstack([cat, cat2])    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of 500um risers with S500 > 52 mJy:  2856\n"
     ]
    }
   ],
   "source": [
    "#select 500um risers with S500 > 52 mJy and S/N >3, these limits can easaly be changed depending on the science goal\n",
    "red_use = (cat['F_RED_MF_SPIRE_250'] < cat['F_RED_MF_SPIRE_350']) & \\\n",
    "          (cat['F_RED_MF_SPIRE_350'] < cat['F_RED_MF_SPIRE_500']) & \\\n",
    "          (cat['F_RED_MF_SPIRE_500'] > 52) & (cat['F_RED_MF_SPIRE_500']/cat['FErr_RED_MF_SPIRE_500'] > 3)\n",
    "cat_red = cat[red_use]\n",
    "print('Number of 500um risers with S500 > 52 mJy: ', np.size(cat_red['RA']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We find a total of 2856 500um risers with S500 > 53 mJy over an area of 900 deg2.\n",
    "### Asboth et al. 2016 found 477 of such SPIRE detected galaxies in the HELMS field. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy GAMA-12\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='GAMA-12_SPIRE500_cat_MF0.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-09T12:09:15' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy GAMA-15\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='GAMA-15_SPIRE500_cat_MF0.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-09T12:14:01' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy HATLAS-NGP\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='HATLAS-NGP_SPIRE500_cat_MF0.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-09T13:08:29' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy HATLAS-SGP\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='HATLAS-SGP_SPIRE500_cat_MF0.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-11T14:18:56' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy SSDF\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='SSDF_SPIRE500_cat_MF0.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-09T12:38:45' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy AKARI-SEP\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_135734_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-29T09:38:39' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'STILVERS' types <class 'str'> and <class 'str'>, choosing STILVERS='3.0-3+' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy Bootes\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_36541_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-09-21T13:49:29' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy CDFS-SWIRE\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_143695_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-09-21T13:51:18' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy COSMOS\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_108980_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-09-21T13:52:24' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy EGS\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_42965_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-29T09:29:29' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy ELAIS-N1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_39933_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-29T09:22:20' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy ELAIS-N2\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_37851_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-29T09:34:59' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy ELAIS-S1\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_141518_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-29T09:29:59' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy HDF-N\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_44911_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-19T13:04:56' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy Lockman-SWIRE\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_24174_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-09-21T13:56:11' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy SA13\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_43372_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-18T10:41:42' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy SPIRE-NEP\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_46546_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-19T13:06:50' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy xFLS\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_40727_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-29T09:35:42' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy XMM-13hr\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_41925_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-06-19T13:06:06' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy XMM-LSS\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_147178_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-09-21T13:59:13' [astropy.utils.metadata]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mJy AKARI-NEP\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "WARNING: MergeConflictWarning: Cannot merge meta key 'EXTNAME' types <class 'str'> and <class 'str'>, choosing EXTNAME='Tile_40949_7_SPIRE_cat.fits' [astropy.utils.metadata]\n",
      "WARNING: MergeConflictWarning: Cannot merge meta key 'DATE-HDU' types <class 'str'> and <class 'str'>, choosing DATE-HDU='2018-09-21T13:46:25' [astropy.utils.metadata]\n"
     ]
    }
   ],
   "source": [
    "# Combining all blind detectet catalogues\n",
    "name = '../dmu22_'+all_names[0]+'/data/dmu22_XID+SPIRE_'+all_names[0]+'_BLIND_Matched_MF.fits'\n",
    "cat_XID=Table.read(name)\n",
    "for j in range(1,np.size(all_names)):\n",
    "    name = '../dmu22_'+all_names[j]+'/data/dmu22_XID+SPIRE_'+all_names[j]+'_BLIND_Matched_MF.fits'\n",
    "    cat2=Table.read(name)\n",
    "    print(cat2['F_BLIND_MF_SPIRE_250'].unit,all_names[j])\n",
    "    cat_XID = vstack([cat_XID, cat2])   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of 500um risers with S500 > 52 mJy:  2728\n",
      "Number of 500um risers with S500 > 52 mJy:  1453\n"
     ]
    }
   ],
   "source": [
    "# selectng 500um risers based on the MF flux density\n",
    "red_use_MF = (cat_XID['F_BLIND_MF_SPIRE_250'] < cat_XID['F_BLIND_MF_SPIRE_350']) & \\\n",
    "          (cat_XID['F_BLIND_MF_SPIRE_350'] < cat_XID['F_BLIND_MF_SPIRE_500']) & \\\n",
    "          (cat_XID['F_BLIND_MF_SPIRE_500'] > 52) & \\\n",
    "          (cat_XID['F_BLIND_MF_SPIRE_500']/cat_XID['FErr_BLIND_MF_SPIRE_500'] > 3)\n",
    "\n",
    "cat_red_MF = cat_XID[red_use_MF]\n",
    "print('Number of 500um risers with S500 > 52 mJy: ', np.size(cat_red_MF['RA']))\n",
    "\n",
    "# selectng 500um risers based on the BLIND XID+ flux density\n",
    "err = cat_XID['F_SPIRE_500'] - cat_XID['FErr_SPIRE_500_l']\n",
    "red_use = (cat_XID['F_SPIRE_250'] < cat_XID['F_SPIRE_350']) & \\\n",
    "          (cat_XID['F_SPIRE_350'] < cat_XID['F_SPIRE_500']) & \\\n",
    "          (cat_XID['F_SPIRE_500'] > 52) & (cat_XID['F_SPIRE_500']/err > 3)\n",
    "cat_red_XID = cat_XID[red_use]\n",
    "print('Number of 500um risers with S500 > 52 mJy: ', np.size(cat_red_XID['RA']))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![HELP LOGO](https://avatars1.githubusercontent.com/u/7880370?s=75&v=4)\n",
    "\n",
    "**Authors**: Steven Duivenvoorden \n",
    "\n",
    "For a full description of the database and how it is organised in to `dmu_products` please the top level [readme](../readme.md).\n",
    "\n",
    "The Herschel Extragalactic Legacy Project, ([HELP](http://herschel.sussex.ac.uk/)), is a [European Commission Research Executive Agency](https://ec.europa.eu/info/departments/research-executive-agency_en)\n",
    "funded project under the SP1-Cooperation, Collaborative project, Small or medium-scale focused research project, FP7-SPACE-2013-1 scheme, Grant Agreement Number 607254.\n",
    "\n",
    "[Acknowledgements](http://herschel.sussex.ac.uk/acknowledgements)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (herschelhelp_internal)",
   "language": "python",
   "name": "helpint"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}