cherrypy database: where did these new fields come from? #614

hppritcha · 2018-01-09T16:28:15Z

We got the cherrypy server running again at AWS and now when I try to submit results I see this HTTP error in the response back from the server

<<<<<<<---------------- Raw Output (Start) ---------------->>>>>>
<!DOCTYPE html PUBLIC
"-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html>
<head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8"></meta>
    <title>500 Internal Server Error</title>
    <style type="text/css">
    #powered_by {
        margin-top: 20px;
        border-top: 2px solid black;
        font-style: italic;
    }

    #traceback {
        color: red;
    }
    </style>
</head>
    <body>
        <h2>500 Internal Server Error</h2>
        <p>The server encountered an unexpected condition which prevented it from fulfilling the request.</p>
        <pre id="traceback">Traceback (most recent call last):
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/env/local/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 670, in respond
    response.body = self.handler()
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/env/local/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 220, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/env/local/lib/python2.7/site-packages/cherrypy/lib/jsontools.py", line 61, in json_handler
    value = cherrypy.serving.request._json_inner_handler(*args, **kwargs)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/env/local/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 60, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/webapp/dispatchers.py", line 360, in POST
    value = self._db.insert_test_run(submit_info['submit_id'], data['metadata'], entry)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/webapp/db_pgv3.py", line 1781, in insert_test_run
    fields, values)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/webapp/db_pgv3.py", line 595, in _select_insert
    cursor.execute( select_stmt, values )
ProgrammingError: column "clck_id" does not exist
LINE 3: ... environment_id = 0 AND result_message_id = 1 AND clck_id = ...
                                                             ^

</pre>
    <div id="powered_by">
      <span>
        Powered by <a href="http://www.cherrypy.org">CherryPy 8.9.0</a>
      </span>
    </div>
    </body>
</html>

anyone know what this clck_id thing is?

The text was updated successfully, but these errors were encountered:

noahv · 2018-01-09T18:09:14Z

Checking on it now. -- Noah

ribab · 2018-01-09T19:38:52Z

Hi Howard,

The issue is that you are using an older Database Schema.

Try and test this patch and see if it makes your code more backwards compatible, you'll have to restart the cherrypy server after applying the change:

diff --git a/server/php/cherrypy/src/webapp/db_pgv3.py b/server/php/cherrypy/src/webapp/db_pgv3.py
index 95cfb9c..16cd433 100644
--- a/server/php/cherrypy/src/webapp/db_pgv3.py
+++ b/server/php/cherrypy/src/webapp/db_pgv3.py
@@ -559,6 +559,9 @@ class DatabaseV3():
     def _select_insert(self, table, table_id, stmt_fields, stmt_values):
         found_id = -1

+        cursor.execute("SELECT * FROM %s LIMIT 0" % table)
+        all_fields_for_table = [d[0] for d in curs.description]
+
         #
         # Build the SELECT and INSERT statements
         #
@@ -567,6 +570,9 @@ class DatabaseV3():

         count = 0
         for field in stmt_fields:
+            if field not in all_fields_for_table:
+                self._logger.debug("WARNING: _select_insert field %s not in table %s" % (field, table))
+                continue
             insert_stmt = insert_stmt + ", " + field

             if count == 0:

Thanks --Ricky

hppritcha · 2018-01-09T19:45:31Z

@jjhursey could you patch the server?

jjhursey · 2018-01-16T17:14:18Z

@hppritcha I applied the patch to the CherryPy server. Please give it another try.

jjhursey · 2018-01-16T19:59:22Z

A couple modifications so far:

+        cursor = self.get_cursor()
+        cursor.execute("SELECT * FROM %s LIMIT 0" % table)
+        all_fields_for_table = [d[0] for d in cursor.description]

Now we get the following error message:

Traceback (most recent call last):
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/env/local/lib/python2.7/site-packages/cherrypy/_cprequest.py", line 670, in respond
    response.body = self.handler()
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/env/local/lib/python2.7/site-packages/cherrypy/lib/encoding.py", line 220, in __call__
    self.body = self.oldhandler(*args, **kwargs)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/env/local/lib/python2.7/site-packages/cherrypy/lib/jsontools.py", line 61, in json_handler
    value = cherrypy.serving.request._json_inner_handler(*args, **kwargs)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/env/local/lib/python2.7/site-packages/cherrypy/_cpdispatch.py", line 60, in __call__
    return self.callable(*self.args, **self.kwargs)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/webapp/dispatchers.py", line 360, in POST
    value = self._db.insert_test_run(submit_info['submit_id'], data['metadata'], entry)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/webapp/db_pgv3.py", line 1788, in insert_test_run
    fields, values)
  File "/mnt/data/mtt.open-mpi.org/mtt/server/php/cherrypy/src/webapp/db_pgv3.py", line 602, in _select_insert
    cursor.execute( select_stmt, values )
TypeError: not all arguments converted during string formatting

The patch emitted these warnings - which were expected

[2018-01-16 19:29:37,341] DEBUG WARNING: _select_insert field clck_id not in table test_run
[2018-01-16 19:29:37,341] DEBUG WARNING: _select_insert field bios_id not in table test_run
[2018-01-16 19:29:37,341] DEBUG WARNING: _select_insert field firmware_id not in table test_run
[2018-01-16 19:29:37,341] DEBUG WARNING: _select_insert field provision_id not in table test_run
[2018-01-16 19:29:37,341] DEBUG WARNING: _select_insert field harasser_id not in table test_run

hppritcha · 2018-02-13T20:44:21Z

okay I think I see what's wrong here. That patch isn't sufficient. We also need to prune the stmt_values otherwise it borks up the cursor.execute thing.

Turns out clients can be generating more fields/values than the database/cherrypy server know about owing to separate MTT based projects contributing code. Fixes open-mpi#614 Signed-off-by: Howard Pritchard <howardp@lanl.gov>

hppritcha added bug Database labels Jan 9, 2018

hppritcha assigned jjhursey Jan 9, 2018

hppritcha assigned hppritcha and unassigned jjhursey Feb 14, 2018

hppritcha mentioned this issue Feb 16, 2018

database: add filter to insert logic #623

Merged

hppritcha closed this as completed in #623 Feb 19, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cherrypy database: where did these new fields come from? #614

cherrypy database: where did these new fields come from? #614

hppritcha commented Jan 9, 2018 •

edited

Loading

noahv commented Jan 9, 2018

ribab commented Jan 9, 2018

hppritcha commented Jan 9, 2018

jjhursey commented Jan 16, 2018

jjhursey commented Jan 16, 2018 •

edited

Loading

hppritcha commented Feb 13, 2018

cherrypy database: where did these new fields come from? #614

cherrypy database: where did these new fields come from? #614

Comments

hppritcha commented Jan 9, 2018 • edited Loading

noahv commented Jan 9, 2018

ribab commented Jan 9, 2018

hppritcha commented Jan 9, 2018

jjhursey commented Jan 16, 2018

jjhursey commented Jan 16, 2018 • edited Loading

hppritcha commented Feb 13, 2018

hppritcha commented Jan 9, 2018 •

edited

Loading

jjhursey commented Jan 16, 2018 •

edited

Loading