F24 and Q8 down/slow. Parse error

[mm98]

Hi,

Thanks for this awesome HA extension!

It looks like F24 and Q8 API endpoints are very slow (randomly) at the moment ("" returned from API's). Which results in all fuel prices will be marked as Unavailable.

This causes multiple timeouts/errors in HA logs:

2022-11-03 08:23:19.212 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform fuelprices_dk is taking over 10 seconds.
2022-11-03 08:24:05.509 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: sensor.fuelprices_dk
2022-11-03 08:24:09.214 ERROR (MainThread) [homeassistant.components.sensor] Setup of platform fuelprices_dk is taking longer than 60 seconds. Startup will proceed without waiting any longer.
2022-11-03 08:24:55.064 ERROR (MainThread) [custom_components.fuelprices_dk.sensor] Unexpected error fetching sensor data: list index out of range
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 164, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/fuelprices_dk/sensor.py", line 30, in async_update_data
    await hass.async_add_executor_job(fuelPrices.refresh)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_api.py", line 151, in refresh
    company.refreshPrices()
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_api.py", line 200, in refreshPrices
    self._products = getattr(self._parser, self._key)(self._url, self._products)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_parsers.py", line 149, in f24
    return self._f24_q8(url, products)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_parsers.py", line 182, in _f24_q8
    json_product = r.json()["Products"][productDict["Index"]]
IndexError: list index out of range

Would it maybe be an option to add the timeout argument to the request.get() and validate, that you get any content back in the response. The status is 200 - but no content.

Fuelprices_DK/custom_components/fuelprices_dk/fuelprices_dk_parsers.py

Lines 192 to 197 in e224e01

	def _get_website(self, url):
	r = self._session.get(url)
	_LOGGER.debug("URL: " + url + " [" + str(r.status_code) + "]")
	if r.status_code != 200:
	return r.status_code
	return r

I guess r.text is valid, but it is empty.

Fuelprices_DK/custom_components/fuelprices_dk/fuelprices_dk_parsers.py

Lines 199 to 201 in e224e01

	def _get_html_soup(self, r, parser="html.parser"):
	if r.text:
	return BS(r.text, parser)

[mediaNen]

Exact same issue for me, since updating to the newest core edition of hass

[J-Lindvig]

It seems like Q8 and F24 is having some issues on their sites with prices.

Since Fuelprices rely on working websites, it will cause an error.

[J-Lindvig]

Hi,

Thanks for this awesome HA extension!

It looks like F24 and Q8 API endpoints are very slow (randomly) at the moment ("" returned from API's). Which results in all fuel prices will be marked as Unavailable.

This causes multiple timeouts/errors in HA logs:

2022-11-03 08:23:19.212 WARNING (MainThread) [homeassistant.components.sensor] Setup of sensor platform fuelprices_dk is taking over 10 seconds.
2022-11-03 08:24:05.509 WARNING (MainThread) [homeassistant.bootstrap] Waiting on integrations to complete setup: sensor.fuelprices_dk
2022-11-03 08:24:09.214 ERROR (MainThread) [homeassistant.components.sensor] Setup of platform fuelprices_dk is taking longer than 60 seconds. Startup will proceed without waiting any longer.
2022-11-03 08:24:55.064 ERROR (MainThread) [custom_components.fuelprices_dk.sensor] Unexpected error fetching sensor data: list index out of range
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 205, in _async_refresh
    self.data = await self._async_update_data()
  File "/usr/src/homeassistant/homeassistant/helpers/update_coordinator.py", line 164, in _async_update_data
    return await self.update_method()
  File "/config/custom_components/fuelprices_dk/sensor.py", line 30, in async_update_data
    await hass.async_add_executor_job(fuelPrices.refresh)
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_api.py", line 151, in refresh
    company.refreshPrices()
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_api.py", line 200, in refreshPrices
    self._products = getattr(self._parser, self._key)(self._url, self._products)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_parsers.py", line 149, in f24
    return self._f24_q8(url, products)
  File "/config/custom_components/fuelprices_dk/fuelprices_dk_parsers.py", line 182, in _f24_q8
    json_product = r.json()["Products"][productDict["Index"]]
IndexError: list index out of range

Would it maybe be an option to add the timeout argument to the request.get() and validate, that you get any content back in the response. The status is 200 - but no content.

Fuelprices_DK/custom_components/fuelprices_dk/fuelprices_dk_parsers.py

Lines 192 to 197 in e224e01

	def _get_website(self, url):
	r = self._session.get(url)
	_LOGGER.debug("URL: " + url + " [" + str(r.status_code) + "]")
	if r.status_code != 200:
	return r.status_code
	return r

I guess r.text is valid, but it is empty.

Fuelprices_DK/custom_components/fuelprices_dk/fuelprices_dk_parsers.py

Lines 199 to 201 in e224e01

	def _get_html_soup(self, r, parser="html.parser"):
	if r.text:
	return BS(r.text, parser)

I am adding the timeout, as I should have done in the first place.... ;-)

The missing data from f24/q8 ("") is because their prices i served by an API and not a webpage.