Apache httpd times out, will not start, when SQLSRV enabled #805
Comments
|
@jamesmontalvo3 Do you have the same problem when trying to run a script that makes use of the sqlsrv or pdo_sqlsrv API from the command line (try php <sometestscript.php>)? Also, what is the output of |
|
Both servers have the same output of To test command line PHP I created the following script: <?php
$conn = 'sqlsrv:server=[redacted] ; Database=[redacted]';
$user = '[redacted]';
$pass = '[redacted]';
$dbh = new PDO( $conn, $user, $pass );
echo "hello\n";On the functioning server I get: On the failing server I get: |
|
I'm not sure if this is significant, but each failed |
|
Since the problem affects command line PHP, I doubt it has anything to do with Apache (though the semaphore leak is strange. Have you seen this link?). To verify that apache is not responsible, could you uninstall Apache from the system and verify that the script produces the same error? Thanks! |
|
I'll uninstall Apache to make sure, but I agree the command line test likely exonerates Apache. Assuming removing it has no effect, what else can I try? Is there some additional logging I can enable? |
|
You can try generating an ODBC trace - in the connection string, set Note that tracing must be set in both the connection string and odbcinst.ini to work. For more information, see this link. |
|
Doing Then I did |
|
The keyword for the connection string is not |
|
Okay to simplify things I moved everything into
<?php
$conn = 'sqlsrv:server=[redacted] ; Database=[redacted]; TraceOn=true; TraceFile=/opt/odbc.log ';
$user = '[redacted]';
$pass = '[redacted]';
$dbh = new PDO( $conn, $user, $pass );
echo "hello\n";
As EDIT: this comment initially had a "could not find driver error" here because I had disabled the MS SQLSRV extensions while troubleshooting. The incorrect error has been removed, and the actual error is below: As There is nothing in |
|
I should have also given this: |
|
The reason for the difference between Still nothing in |
|
Hi @jamesmontalvo3 , @david-puglielli is not in the office today. Anyway, with an error message like |
|
@yitam I'm not sure what you're suggesting. Can you provide an edited version of Keep in mind that simply trying to load |
|
@jamesmontalvo3, @yitam is referring to the |
|
I'm not sure how a network issue could prevent Apache from loading, but will give it a try. I have not used those before, and a quick search didn't show anything. I'll keep looking, but can you provide some docs? FYI, I changed <?php
$server = "[redacted]";
$db = "[redacted]";
$table = "[redacted]";
$conn = "sqlsrv:server=$server ; Database=$db; TraceOn=true; TraceFile=/opt/odbc.log ";
$user = '[redacted]';
$pass = '[redacted]';
$conn = sqlsrv_connect( $server, [ "Database" => $db, "UID" => $user, "PWD" => $pass] );
if( $conn === false ) {
die( print_r( sqlsrv_errors(), true));
}
$stmt = sqlsrv_query( $conn, "SELECT * FROM $table LIMIT 1", [] );
print_r( sqlsrv_fetch_array ( $stmt ) );
echo "hello\n";Which gave the output: To try to isolate the Apache startup problem, I directly ran Each pair of [1] These lines repeated every ~1 second: [2] These lines were repeated every ~10 seconds: |
|
Okay I think I figured out what you wanted from Created And SELECT * FROM [redacted] LIMIT 2
And ran |
|
OK, that strongly suggests a network issue, because |
|
But why would that prevent Apache from starting? |
|
I don't know as I am not an Apache expert, but there is definitely a problem accessing your SQL Server and it's not clear that the Apache issue is even related to that. In any case, I would start by making sure the connection to the server works. The network error code 0x2AF9 often occurs when the wrong port is specified in the connection string. You can double check the port number by issuing the following query: This thread has more information on this error. |
|
You were right: there was a firewall rule in place preventing me from reaching that database server. This server had previously connected using PHP 5.6 and the mssql package that came with it...so that rule must have been recently added, conflating this issue. I resolved the network issue and am able to connect in a command line PHP script. That did not resolve the Apache issue, however. |
|
Hi @jamesmontalvo3 , @david-puglielli is not in today. In the meantime, please check if you have more than one php versions and/or php.ini files in your env (they may be conflicting one another). Besides, I suppose the same steps that you have followed worked in CentOS but not Red Hat? I'm not entirely sure why, but this article about Red Hat subscriptions or collections might be helpful. |
|
@yitam good idea to try it on another RedHat box. I booted a RedHat install and tried it: no issues. I've posted this issue on the Apache mailing list, and will report back here any relevant info. If you can think of anything else I'd really appreciate it. Thank you so much for the support thus far! |
|
That's good news, @jamesmontalvo3 , and you're welcome. |
|
Update: If I start Apache httpd manually ( |
|
Posted question on Server Fault: https://serverfault.com/questions/919326/apache-wont-start-with-systemd Also, here's the apache users mailing list archive link: http://mail-archives.apache.org/mod_mbox/httpd-users/201807.mbox/%3CCANkjp2rF2beFWWaEpLSouCOc+txzdGh0rihLYz4s4-uWQmOoOg@mail.gmail.com%3E |
|
Thanks @jamesmontalvo3 for the info. Didn't you say that on the other RedHat box there was no issue? Or you found out the same issue happened in both? |
|
No issues on a RedHat VM I created in VirtualBox. |
|
In that case, it sounds like some configuration issues particular to your settings / environment. As I pointed out before, is it possible that this has something to do with subscriptions? |
|
I'm investigating differences in installed packages from one server to the other. There may be something installed with the base install of the servers that is either helping or hindering. Subscriptions isn't likely an issue, unless you're saying some subscription needs to be in place for SQLSRV. With SQLSRV enabled I get the error. With it disabled (just removing |
|
No, @jamesmontalvo3 , not that we are aware of. So far we haven't received any issue that's related to what you've experienced. |
|
As per this comment [1] I can get httpd to start with systemctl if the timeout is extended enough. So now I just need to figure out why having PHP SQLSRV enabled causes so much lag. Thoughts on how to identify the issue? [1] https://serverfault.com/questions/919326/apache-wont-start-with-systemd#comment1190872_919581 |
|
I believe I found the problem. I believe that it's hanging when looking for the file |
|
That sounds like the probable cause. But that's a non-standard name and location for the odbcinst.ini file. I'm guessing that you compiled unixodbc from source - is that correct? What configure options did you use? What is the output of There is probably a configure option or environment variable that governs where Apache/unixodbc/ODBC driver looks for odbcinst.ini, but I will have to get back to you on that. |
|
I did not compile from source. I followed the directions linked off of the Microsoft/msphpsql project. See my adaptation of the instructions at the bottom of this comment. I agree that
for root: I've tried overriding defaults by creating export ODBCINI=/etc/odbc.ini
export ODBCINSTINI=/etc/odbcinst.ini
export ODBCSYSINI=/etcFYI: My adaptation of the install instructions, to be handled with Ansible: - name: Ensure prerequisites for sqlsrv in place
yum:
name: "{{item}}"
state: installed
with_items:
- re2c
- gcc-c++
# Install ODBC driver
- name: install mssql-server repo (CentOS, RedHat)
get_url:
url: https://packages.microsoft.com/config/rhel/7/prod.repo
dest: /etc/yum.repos.d/mssql-release.repo
when: ansible_distribution in ['CentOS', 'RedHat']
- name: Ensure conflicting ODBC drivers removed
yum:
name: "{{item}}"
state: absent
with_items:
- unixODBC-utf16
- unixODBC-utf16-devel
- name: install MS ODBC driver package
yum:
name: msodbcsql17
state: latest
environment:
ACCEPT_EULA: 'y'
notify:
- restart apache
- name: install ODBC driver devel package
yum:
name: unixODBC-devel
state: latest |
|
I added the following to However, I'm getting the error:
If I stop httpd, then start it manually via |
|
The output of Please double-check that |
|
I've played around with The contents of I tried modifying |
|
Yes @jamesmontalvo3 this is weird (see related issue #799, another Red Hat user ). While @david-puglielli is investigating, ODBC team suggested you do an strace to find out what went wrong. Apparently, unixOBDC failed to find odbcinst.ini in your system, and regarding your env variables, this man page may be helpful. Meanwhile, please compare your settings to your other Red Hat vm (in which everything works as is). |
|
When checking user Apache's I've been unsuccessful trying to get systemd to output what it thinks Apache's So I changed the end of I'd like to spend a little time making sure there are no other issues associated with this, but otherwise I think this probably closes the issue. There's definitely some weirdness going on with the Thank you so much for your help! |
|
Congrats @jamesmontalvo3 ! This is very helpful info, and we will put in our FAQ for other users. |
|
@yitam the significant settings difference is that the failing system has My guess is that all Apache-based systems on RedHat/CentOS and loading with systemd are looking here, but most systems quickly see that the file doesn't exist and move on. This one just lags A LOT looking for the file. |
|
Any update, @jamesmontalvo3 ? I'm closing this due to inactivity. Please feel free to reopen if you have any other question. |
|
All seems to be working. Concur with closing. Thanks for the help! |
When
extension=sqlsrv.soandextension=pdo_sqlsrv.soare enabled, Apache httpd will timeout and not start. This is occurring on one server, but I cannot replicate on two other servers. All servers are managed through the same Ansible scripts and should be very consistently built. I'm having trouble determining what is causing this issue.If I run
sudo systemctl restart httpd, then after 3 minutes (exactly), this error occurs:Removing file
/etc/php.d/20-sqlsrv.iniresults in the same problem. Reinstating/etc/php.d/20-sqlsrv.iniand instead removing/etc/php.d/30-pdo_sqlsrv.iniresults in the same problem. Removing both allows Apache httpd to start normally.Below I make reference to the "Failing system" which is showing this error and the "Comparison system" which is not seeing the error. I've also run the same setup on a VirtualBox CentOS setup and haven't seen the problem. I'm at a loss for how to determine why this one system will not work.
PHP Driver version or file name
php --re sqlsrv | head -1gives:Failing system:
Extension [ <persistent> extension #43 sqlsrv version 5.2.0 ] {Comparison system:
Extension [ <persistent> extension #43 sqlsrv version 5.2.0 ] {php -re pdo_sqlsrv | head -1gives:Failing system:
Extension [ <persistent> extension #56 pdo_sqlsrv version 5.2.0 ] {Comparison system:
Extension [ <persistent> extension #56 pdo_sqlsrv version 5.2.0 ] {SQL Server version
N/A. Can't even get Apache to start. This exact setup has worked on Microsoft SQL Server 2016 without issue.
Client operating system
Failing system: Red Hat Enterprise Linux Server release 7.5 (Maipo)
Comparison system: CentOS Linux release 7.5.1804 (Core)
PHP version
Failing system: PHP 7.0.30 (cli) (built: Apr 26 2018 13:30:55) ( NTS )
Comparison system: PHP 7.0.30 (cli) (built: Apr 26 2018 13:30:35) ( NTS )
Microsoft ODBC Driver version
Doing
yum info msodbcsql17yields the following:Failing system:
Comparison system:
Table schema
N/A
Problem description
See above.
Expected behavior and actual behavior
Expected: Apache starts
Actual: Apache times out and fails
Repro code or steps to reproduce
Installation is managed by a set of Ansible scripts. Specifically:
Relevant logs
After running
echo $(date) && sudo systemctl restart httpdbelow are the relevant log files after the echoed time stampMon Jun 25 11:46:25 CDT 2018. Estimated failure time at 11:49:25 since it's been timing out 3 minutes after runningrestart httpd.journalctl -xe output
/var/log/secure
php error log
No entries from this time period
/var/log/httpd/error_logNote: Apache
LogLevelset todebug.More info
yum install httpd-devel, PHP 7.0 from IUS repositoryThe text was updated successfully, but these errors were encountered: