TheTVDB mirrors.xml returning error 503

[Omertron]

Original issue 1589 created by Omertron on 2010-09-09T05:48:49.000Z:

When refreshing my Jukebox, I see the following error:

ERROR: TheTVDB API -> Server returned HTTP response code: 503 for URL: http://www.thetvdb.com/api/2805AD2873519EC5/mirrors.xml

However the referenced URL returns a valid page of XML.

What version of YAMJ are you using? On what operating system?
rvn1770, but also fails with rvn1761. OS is Win7 64-bit.

What device are you using YAMJ on? Networked Media Tank, PlayonHD, Other?
PCH C-200.

[Omertron]

Comment #1 originally posted by Omertron on 2010-09-10T18:47:49.000Z:

The likelihood of getting the 503 seems to increase with a higher number of download threads, eg. 64

mjb.MaxThreadsProcess=64
mjb.MaxThreadsDownload=0

Even though thetvdb is limited to one connection by:

mjb.MaxDownloadSlots=._=2,.imdb.=2,.google.=5,.yahoo.=5,.themoviedb.=1,.thetvdb.=1,._apple.*=1

Is there an issue (too many concurrent connections?) when accessing this mirror file while processing a large number of threads?

Not that I can access the tvdb today even when using a low number of threads - I'm just getting this now (I'm sure it will go away eventually):

[TheTVDB API] Episode error: com.sun.org.apache.xerces.internal.impl.io.Malforme
dByteSequenceException: Invalid byte 1 of 1-byte UTF-8 sequence.

[Omertron]

Comment #2 originally posted by Omertron on 2010-09-10T21:02:17.000Z:

Confirmed I would suggest not using that many threads as a workaround :)

[Omertron]

Comment #3 originally posted by Omertron on 2010-09-11T09:05:20.000Z:

64 is a huge number of processing threads... I would never make it more than 4/8.

The framework for limiting the connections can not control what's happening with the actual socket connection when leaving the call. Java does NOT specify what it does when closing a connection; very likely that it caches them (depending on implementation).
Looking at TheTvDBPlugin, doscan() and scanTVShowTitles(), there are lots of calls inside enterIO, leaveIO block. I don't know which of those calls open new connections and if they try to close them... It is possible that they open many connections that remain cached; I can also assume that the Java implementation caches connections per thread. Together that could give a high number of open connections...
See http://thetvdb.com/wiki/index.php?title=Programmers_API :
"Please avoid making more API calls than are necessary to retrieve the information you need."

The first workaround is indeed to use a small number of threads; but the real fix could be inside TheTvDB API...
If look at WebBrowser.request(), I tried to explicitly close the http connection.
When having some time, I will try to look at the source.

[Omertron]

Comment #4 originally posted by Omertron on 2010-09-11T14:43:06.000Z:

Stuart, can you try my changes to thetvdb?
I had no chance to test it, so please review... although the changes are quite understandable.
I tried to use only 1 webbrowser instance (the class duplicates the yamj one...) and to close the used connections.

[Omertron]

Comment #5 originally posted by Omertron on 2010-09-11T15:44:07.000Z:

Still getting the same errors, though they appear quicker...

{16:06:06} Processing new video : Lost [Season 3]
ERROR: TheTVDB API -> Server returned HTTP response code: 503 for URL: http://www.thetvdb.com/api/[APIKEY]/mirrors.xml
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
ERROR: TheTVDB API -> Server returned HTTP response code: 503 for URL: http://www.thetvdb.com/api/[APIKEY]/mirrors.xml[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en

[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
ERROR: TheTVDB API -> Server returned HTTP response code: 503 for URL: http://www.thetvdb.com/api/[APIKEY]/mirrors.xml[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en

[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en
[TheTVDB API] Series error: no protocol: null/api/GetSeries.php?seriesname=Lost&language=en

[Omertron]

Comment #6 originally posted by Omertron on 2010-09-11T17:26:18.000Z:

It looks like the xmlMirror variable is null (in constructor!).
You have to debug and see why/when can that happen; the first error is thrown in Mirrors() constructor, later everything is a just a consequence...
It can be that thetvdb doesn't like multiple connections, so we might need to adjust the processing... now there is one instance per thread and each thread queries thetvdb on start.
I added a few simple message, to see when it happens...

I also saw that webbrowser variable is never really used in TheTVDB; connections are opened via XMLHelper, which has it's own webbrowser; it is only used in setProxy(), so proxies can't work (if anyone noticed...).

[Omertron]

Comment #7 originally posted by Omertron on 2010-09-30T22:18:21.000Z:

Currently I can only scan my jukebox with a single processing thread and single download thread, which is a shame as I have a quad core box with 2MB connection so in theory can quite happily run YAMJ with dozens of processing threads.

As things stand if YAMJ needs to access the TVDB the benefits of multi-threading go out the window - the TVDB API just won't work reliably with more than a single thread. In fact even when scanning with only a single thread, accessing the TVDB API is still flaky (see issue 1615).

[Omertron]

Comment #8 originally posted by Omertron on 2010-10-01T05:50:56.000Z:

I run with 8 processing and 4 downloading threads on old jukeboxes (re-runs) and get very few of these errors.

I'm re-writing theTVDb.com API to use a different method of parsing the XML and hope that's going to fix these issues

[Omertron]

Comment #9 originally posted by Omertron on 2010-10-01T09:47:27.000Z:

Thanks Stuart - let me know if you think I can help with testing.

[Omertron]

Comment #10 originally posted by Omertron on 2010-10-05T08:15:12.000Z:

This should be fixed now.

[Omertron]

Comment #11 originally posted by Omertron on 2010-10-05T09:43:35.000Z:

Looking good with r1819 - many thanks.