{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ⌚ Latest Data\n",
    "\n",
    "Herbie can help you find the latest model data available with `HerbieLatest` and `HerbieWait`. These are simple helper functions that loops until Herbie finds data.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "from herbie import HerbieLatest, HerbieWait"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `HerbieLatest`\n",
    "\n",
    "Herbie can find the latest model grid available. It does this by checking the for data in the most recent time, and then steps back in time a model cycle and checks for that. By default, if data is not found in four attempts, it will return an error.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Found ┊ model=hrrr ┊ \u001b[3mproduct=sfc\u001b[0m ┊ \u001b[38;2;41;130;13m2024-Jan-19 19:00 UTC\u001b[92m F06\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3mGRIB2 @ aws\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3mIDX @ aws\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\u001b[48;2;255;255;255m\u001b[38;2;136;33;27m▌\u001b[0m\u001b[38;2;12;53;118m\u001b[48;2;240;234;210m▌\u001b[38;2;0;0;0m\u001b[1mHerbie\u001b[0m HRRR model \u001b[3msfc\u001b[0m product initialized \u001b[38;2;41;130;13m2024-Jan-19 19:00 UTC\u001b[92m F06\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3msource=aws\u001b[0m"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H = HerbieLatest(model=\"hrrr\", fxx=6)\n",
    "H"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "✅ Found ┊ model=gfs ┊ \u001b[3mproduct=pgrb2.0p25\u001b[0m ┊ \u001b[38;2;41;130;13m2024-Jan-19 18:00 UTC\u001b[92m F00\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3mGRIB2 @ aws\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3mIDX @ aws\u001b[0m\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "\u001b[48;2;255;255;255m\u001b[38;2;136;33;27m▌\u001b[0m\u001b[38;2;12;53;118m\u001b[48;2;240;234;210m▌\u001b[38;2;0;0;0m\u001b[1mHerbie\u001b[0m GFS model \u001b[3mpgrb2.0p25\u001b[0m product initialized \u001b[38;2;41;130;13m2024-Jan-19 18:00 UTC\u001b[92m F00\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3msource=aws\u001b[0m"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H = HerbieLatest(model=\"gfs\")\n",
    "H"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "💔 Did not find ┊ model=hrrr ┊ \u001b[3mproduct=sfc\u001b[0m ┊ \u001b[38;2;41;130;13m2024-Jan-19 19:00 UTC\u001b[92m F100\u001b[0m\n",
      "💔 Did not find ┊ model=hrrr ┊ \u001b[3mproduct=sfc\u001b[0m ┊ \u001b[38;2;41;130;13m2024-Jan-19 18:00 UTC\u001b[92m F100\u001b[0m\n",
      "💔 Did not find ┊ model=hrrr ┊ \u001b[3mproduct=sfc\u001b[0m ┊ \u001b[38;2;41;130;13m2024-Jan-19 17:00 UTC\u001b[92m F100\u001b[0m\n",
      "💔 Did not find ┊ model=hrrr ┊ \u001b[3mproduct=sfc\u001b[0m ┊ \u001b[38;2;41;130;13m2024-Jan-19 16:00 UTC\u001b[92m F100\u001b[0m\n"
     ]
    },
    {
     "ename": "TimeoutError",
     "evalue": "Herbie did not find data for the latest time: ║HERBIE╠ HRRR:sfc",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTimeoutError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[5], line 3\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[38;5;66;03m# This will error because there will never been\u001b[39;00m\n\u001b[1;32m      2\u001b[0m \u001b[38;5;66;03m# a lead time of 100 hours.\u001b[39;00m\n\u001b[0;32m----> 3\u001b[0m H \u001b[38;5;241m=\u001b[39m \u001b[43mHerbieLatest\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmodel\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mhrrr\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mfxx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m100\u001b[39;49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/GITHUB/Herbie/herbie/latest.py:53\u001b[0m, in \u001b[0;36mHerbieLatest\u001b[0;34m(model, priority, periods, **kwargs)\u001b[0m\n\u001b[1;32m     50\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m H\u001b[38;5;241m.\u001b[39mgrib:\n\u001b[1;32m     51\u001b[0m         \u001b[38;5;28;01mreturn\u001b[39;00m H\n\u001b[0;32m---> 53\u001b[0m \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTimeoutError\u001b[39;00m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mHerbie did not find data for the latest time: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mH\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n",
      "\u001b[0;31mTimeoutError\u001b[0m: Herbie did not find data for the latest time: ║HERBIE╠ HRRR:sfc"
     ]
    }
   ],
   "source": [
    "# This will error because there will never been\n",
    "# a lead time of 100 hours.\n",
    "H = HerbieLatest(model=\"hrrr\", fxx=100)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `HerbieWait`\n",
    "\n",
    "If you want to wait for model data to become available in real time, Herbie will use a while loop to wait for data.\n",
    "\n",
    "In this example, I've changed the default wait time and interval just to demonstrate. (The error in this is expected.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n"
     ]
    },
    {
     "ename": "TimeoutError",
     "evalue": "Herbie did not find data in time: ║HERBIE╠ RAP:awp130pgrb",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mTimeoutError\u001b[0m                              Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[14], line 2\u001b[0m\n\u001b[1;32m      1\u001b[0m run \u001b[38;5;241m=\u001b[39m pd\u001b[38;5;241m.\u001b[39mTimestamp(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnow\u001b[39m\u001b[38;5;124m\"\u001b[39m, tz\u001b[38;5;241m=\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mutc\u001b[39m\u001b[38;5;124m\"\u001b[39m)\u001b[38;5;241m.\u001b[39mfloor(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m1h\u001b[39m\u001b[38;5;124m'\u001b[39m)\u001b[38;5;241m.\u001b[39mreplace(tzinfo\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mNone\u001b[39;00m)\n\u001b[0;32m----> 2\u001b[0m H \u001b[38;5;241m=\u001b[39m \u001b[43mHerbieWait\u001b[49m\u001b[43m(\u001b[49m\u001b[43mrun\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mrun\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mmodel\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mrap\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mproduct\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mawp130pgrb\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mwait_for\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m10s\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mcheck_interval\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43m1s\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mfxx\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/mapwall_dev/herbie-dev/herbie/latest.py:110\u001b[0m, in \u001b[0;36mHerbieWait\u001b[0;34m(run, model, priority, wait_for, check_interval, **kwargs)\u001b[0m\n\u001b[1;32m    108\u001b[0m     \u001b[38;5;66;03m# Error out if timeout is exceeded\u001b[39;00m\n\u001b[1;32m    109\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m (pd\u001b[38;5;241m.\u001b[39mTimestamp(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mnow\u001b[39m\u001b[38;5;124m\"\u001b[39m) \u001b[38;5;241m-\u001b[39m timer) \u001b[38;5;241m>\u001b[39m\u001b[38;5;241m=\u001b[39m pd\u001b[38;5;241m.\u001b[39mTimedelta(wait_for):\n\u001b[0;32m--> 110\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mTimeoutError\u001b[39;00m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mHerbie did not find data in time: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mH\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m    112\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m Herbie(run, model\u001b[38;5;241m=\u001b[39mmodel, priority\u001b[38;5;241m=\u001b[39mpriority, \u001b[38;5;241m*\u001b[39m\u001b[38;5;241m*\u001b[39mkwargs)\n",
      "\u001b[0;31mTimeoutError\u001b[0m: Herbie did not find data in time: ║HERBIE╠ RAP:awp130pgrb"
     ]
    }
   ],
   "source": [
    "run = pd.Timestamp(\"now\", tz=\"utc\").floor('1h').replace(tzinfo=None)\n",
    "H = HerbieWait(run=run, model=\"rap\", product=\"awp130pgrb\", wait_for=\"10s\", check_interval=\"1s\", fxx=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here's an example that takes a while to run, demonstrating that it works!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "💔 Did not find ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m\n",
      "✅ Found ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3mGRIB2 @ nomads\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3mIDX @ nomads\u001b[0m\n",
      "✅ Found ┊ model=rap ┊ \u001b[3mproduct=awp130pgrb\u001b[0m ┊ \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3mGRIB2 @ nomads\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3mIDX @ nomads\u001b[0m\n"
     ]
    }
   ],
   "source": [
    "H = HerbieWait(run=run, model=\"rap\", product=\"awp130pgrb\", wait_for=\"2h\", check_interval=\"120s\", fxx=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\u001b[48;2;255;255;255m\u001b[38;2;136;33;27m▌\u001b[0m\u001b[38;2;12;53;118m\u001b[48;2;240;234;210m▌\u001b[38;2;0;0;0m\u001b[1mHerbie\u001b[0m RAP model \u001b[3mawp130pgrb\u001b[0m product initialized \u001b[38;2;41;130;13m2024-May-21 17:00 UTC\u001b[92m F00\u001b[0m ┊ \u001b[38;2;255;153;0m\u001b[3msource=nomads\u001b[0m"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "H"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "herbie-dev",
   "language": "python",
   "name": "herbie-dev"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}