加入mongodb审核，重写mongodb查询等 #900

fancy-lee · 2020-09-30T11:15:49Z

1.加入mongodb审核，
2.重写mongodb查询支持原生查询语法
3.查询编辑框加入缓存，避免刷新丢失
4.mongo审核时关闭备份下拉框等
5.优化mongodb查询显示效果

LeoQuote

感谢提交pr

单元测试一定要过

其他的事情看评论，另外不要重提pr，请直接在pr分支上提交，如果新开PR我就不再参与了。

sql/engines/mongo.py

sql/engines/mysql.py

sql/templates/sqlsubmit.html

sql/engines/mongo.py

LeoQuote · 2020-09-30T16:24:38Z

说实话这是我们第一个有自己的审核机制的engine，mysql的是调外部审查的，关于这些希望还是保持和之前一样有配套的单元测试，不然后续没人敢改，不好维护的。

fancy-lee · 2020-10-01T04:10:11Z

说实话这是我们第一个有自己的审核机制的engine，mysql的是调外部审查的，关于这些希望还是保持和之前一样有配套的单元测试，不然后续没人敢改，不好维护的。

我把mysql的部分修改回来吧，第一次提pr不太熟悉流程，请多指教

LeoQuote · 2020-10-01T09:14:48Z

不只是mysql的事情，mongo 的engine也是有单元测试的，mongo 的测试也得过

fancy-lee · 2020-10-01T16:45:38Z

不只是mysql的事情，mongo 的engine也是有单元测试的，mongo 的测试也得过
请问Codacy/PR Quality Review 中的提示都得通过吗？Travis CI 测试mongo这块没有通过，但没有看出来问题，请帮忙看一下

LeoQuote · 2020-10-02T01:58:22Z

checks 里travis 里可以点 this build 链接，然后你就可以看到日志

LeoQuote · 2020-10-02T03:32:17Z

我们的engine相关的测试除了mysql外，应该都是patch掉具体的调用，然后看内部逻辑的 #42 参考这个

codecov · 2020-10-02T15:57:13Z

Codecov Report

Merging #900 into master will decrease coverage by 0.96%.
The diff coverage is 49.75%.

@@            Coverage Diff             @@
##           master     #900      +/-   ##
==========================================
- Coverage   78.27%   77.31%   -0.97%     
==========================================
  Files          77       76       -1     
  Lines       11030    11503     +473     
==========================================
+ Hits         8634     8893     +259     
- Misses       2396     2610     +214

Impacted Files	Coverage Δ
sql/query.py	`70.74% <ø> (ø)`
sql/engines/mongo.py	`47.70% <46.26%> (+9.32%)`	⬆️
sql/engines/tests.py	`99.60% <100.00%> (+0.01%)`	⬆️
sql/utils/human_time.py

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 20462a1...d98804e. Read the comment docs.

fancy-lee · 2020-10-02T16:19:28Z

我们的engine相关的测试除了mysql外，应该都是patch掉具体的调用，然后看内部逻辑的 #42 参考这个

请问Codacy/PR Quality Review中所有的问题都要修改吗？新加的每个方法都要写单元测试？

LeoQuote · 2020-10-02T16:42:04Z

👏 恭喜测试全过了，你可以看到 codecov 的覆盖率报告，里面可以看到哪些覆盖到哪些没覆盖，没有覆盖到的方法建议都写一下测试。

quality 的报告可以做参考，主要是风格方面的，也建议遵照改一下。

sql/templates/sqlsubmit.html

sql/templates/sqlquery.html

LeoQuote · 2020-10-06T08:31:00Z

一是上面的提到的事情, 建议都看一下.
另外建议按照engine 设计的来做, 也就是 engine 中的 query_check 方法用来做语句的初检, 如果此时语句为空, 可以返回 bad check , 并且报错, 这部分是有现成的前端返回的, 不需要你在几乎所有的方法里都要校验语句是否为空, 即使校验也是在 query_check 里校验

Archery/sql/query.py

Lines 58 to 63 in 20462a1

    
           query_check_info = query_engine.query_check(db_name=db_name, sql=sql_content) 
        
           if query_check_info.get('bad_query'): 
        
               # 引擎内部判断为 bad_query 
        
               result['status'] = 1 
        
               result['msg'] = query_check_info.get('msg') 
        
               return HttpResponse(json.dumps(result), content_type='application/json')

然后用 filter_sql 做查询时的语句改写

Archery/sql/query.py

Line 84 in 20462a1

sql_content = query_engine.filter_sql(sql=sql_content, limit_num=limit_num)

LeoQuote · 2020-10-06T08:34:22Z

sql/engines/mongo.py

+        return self.error_info
+
+
+class JsonDecoder:


这个 decoder , 如果有现成的用现成的, 没有现成的所有方法尽量都加上测试. 不然除了作者基本不敢动

目前还没有找到现成好用的decoder,这个类可以单独拿出来做一下测试的，我这边本地的测试环境配置有点问题，辛苦你加一下？

我没有办法帮你加测试环境呀，你得自己看是什么问题然后解决

我没有办法帮你加测试环境呀，你得自己看是什么问题然后解决

它只是一个独立的工具类，它的主要工功能是对语句中db.col.find({"id":123})的{}部分解析成dict,它可以单独拿出来做测试和修改，应该不会像你说的"除了作者不敢动”这么严重，因为我本地测试环境有问题，只能push上来做测试，所以希望你帮忙加一下测试

sql/query.py

Co-authored-by: Leo Q <LeoQuote@users.noreply.github.com>

fancy-lee · 2020-10-06T13:53:53Z

一是上面的提到的事情, 建议都看一下.
另外建议按照engine 设计的来做, 也就是 engine 中的 query_check 方法用来做语句的初检, 如果此时语句为空, 可以返回 bad check , 并且报错, 这部分是有现成的前端返回的, 不需要你在几乎所有的方法里都要校验语句是否为空, 即使校验也是在 query_check 里校验

Archery/sql/query.py

Lines 58 to 63 in 20462a1

query_check_info = query_engine.query_check(db_name=db_name, sql=sql_content)

if query_check_info.get('bad_query'):

# 引擎内部判断为 bad_query

result['status'] = 1

result['msg'] = query_check_info.get('msg')

return HttpResponse(json.dumps(result), content_type='application/json')

然后用 filter_sql 做查询时的语句改写

Archery/sql/query.py

Line 84 in 20462a1

sql_content = query_engine.filter_sql(sql=sql_content, limit_num=limit_num)

我把语句为空的这个校验去了

LeoQuote

就这一个地方改了就可以m了，单元测试什么的没条件后续再补吧

如果新的审核需要安装mongo client ，也需要改一下 dockerfile

LeoQuote · 2020-10-07T09:32:44Z

sql/engines/mongo.py

-        self.db_name = self.db_name or 'admin'
-        conn = pymongo.MongoClient(self.host, self.port, authSource=self.db_name, connect=True, connectTimeoutMS=10000)
+        self.db_name = db_name or 'admin' #=======这里要注意一下
+        self.db_name = 'admin'


这个按照讨论改一下吧

fancy-lee · 2020-10-08T01:46:53Z

就这一个地方改了就可以m了，单元测试什么的没条件后续再补吧

如果新的审核需要安装mongo client ，也需要改一下 dockerfile

在容器中下载：https://fastdl.mongodb.org/linux/mongodb-linux-x86_64-rhel70-3.6.20.tgz，解压后在bin目录下把mongo命令复制到指定目录，然后配置下环境变量或软链接，保证在任何地方能执行mongo命令，最后删除原来的压缩包和解压目录。生成一个新的基础镜像就可以了

LeoQuote · 2020-10-09T10:19:22Z

https://github.com/fancy-lee/Archery/blob/archery-fancy/src/docker/Dockerfile-base 帮忙改一下 dockerfile 吧, 在最后面加上你说的那些步骤就可以了.

fancy-lee · 2020-10-11T06:34:06Z

https://github.com/fancy-lee/Archery/blob/archery-fancy/src/docker/Dockerfile-base 帮忙改一下 dockerfile 吧, 在最后面加上你说的那些步骤就可以了.

已经加完了

LeoQuote

👏 LGTM

hhyo · 2020-10-12T15:50:21Z

感谢两位

hhyo · 2020-10-16T00:22:43Z

sql/templates/sqlsubmit.html

@@ -125,6 +127,10 @@
                                <input id="btn-submitsql" type="button" class="btn btn-success" value="SQL提交"/>
                            </div>
                        </div>
+                       <div class="text-info">
+                            <li>提交的语句请以分号结束</li>
+                            <li>[指定执行时间]用于定时任务，不是必选项</li>


执行时间可能理解有误，这个并不是用于定时，是用于错峰执行

hhyo · 2020-10-16T00:25:45Z

sql/templates/sqlsubmit.html

@@ -266,6 +272,7 @@ <h4 class="modal-title text-danger">提交信息确认</h4>
        $("#instance_name").change(function () {
            var optgroup = $('#instance_name :selected').parent().attr('label');
            $("#div-backup").show();
+            if (optgroup != "MySQL") {$("#div-backup").hide(); $("#is_backup").val("False");}


目前Oracle也支持备份，这块后面我处理一下

1.如果查询的实例名和db_name，tab_name不为空，但是sqlcontent为空时会自动创建生成一条查询语句

d0c9672

LeoQuote requested changes Sep 30, 2020

View reviewed changes

sql/engines/mongo.py Outdated Show resolved Hide resolved

sql/engines/mysql.py Outdated Show resolved Hide resolved

sql/templates/sqlsubmit.html Outdated Show resolved Hide resolved

sql/engines/mongo.py Show resolved Hide resolved

sql/engines/mongo.py Show resolved Hide resolved

fancy-lee requested a review from LeoQuote October 1, 2020 03:55

修改pr check

fb46ee6

fancy-lee added 5 commits October 1, 2020 18:07

pr修改2

85d0acc

pr修改3

1f33582

修改测试mongo查询为mongodb原生查询语法

539e0a2

修改单元测试的db_name

6065a71

优化mongodb查询显示

333c2e9

提交mongo-test_query_check()修改入参

c6efce4

fancy-lee added 4 commits October 2, 2020 23:08

修改单元测试test_query_check

7a89828

修改单元测试test_query_check

f877f5a

修改单元测试test_query_check

8118f59

还原query.py

c63d84f

fancy-lee added 7 commits October 4, 2020 19:50

提交单元测试及mongodb查询的优化

fe665e8

提交单元测试及mongodb查询的优化

e77f80a

提交单元测试及mongodb查询的优化

d337d52

提交单元测试及mongodb查询的优化

c3670d0

提交单元测试及mongodb查询的优化

d092600

提交单元测试及mongodb查询的优化

769fe5b

提交单元测试及mongodb查询的优化

80c8552

LeoQuote reviewed Oct 6, 2020

View reviewed changes

sql/templates/sqlsubmit.html Outdated Show resolved Hide resolved

LeoQuote reviewed Oct 6, 2020

View reviewed changes

sql/templates/sqlquery.html Outdated Show resolved Hide resolved

LeoQuote reviewed Oct 6, 2020

View reviewed changes

sql/templates/sqlquery.html Outdated Show resolved Hide resolved

LeoQuote reviewed Oct 6, 2020

View reviewed changes

sql/query.py Show resolved Hide resolved

fancy-lee and others added 2 commits October 6, 2020 21:36

还原sqlsubmit.html

3235be9

Update sql/templates/sqlsubmit.html

9ea7b5a

Co-authored-by: Leo Q <LeoQuote@users.noreply.github.com>

fancy-lee added 4 commits October 6, 2020 21:58

修改版本合并请求

eb81b0e

修改版本合并请求

6cda84e

解决冲突

4447406

还原sqlsubmit.html

e20ff03

LeoQuote requested changes Oct 7, 2020

View reviewed changes

fancy-lee added 2 commits October 11, 2020 14:20

Update Dockerfile-base

fffc467

Update Dockerfile-base

d98804e

LeoQuote approved these changes Oct 12, 2020

View reviewed changes

LeoQuote requested a review from hhyo October 12, 2020 06:26

hhyo approved these changes Oct 12, 2020

View reviewed changes

LeoQuote merged commit 3343a70 into hhyo:master Oct 13, 2020

hhyo reviewed Oct 16, 2020

View reviewed changes

hhyo mentioned this pull request Oct 17, 2020

[ 功能建议 ]SQL上线支持MongoDB #494

Closed

hhyo pushed a commit that referenced this pull request Oct 31, 2020

加入mongodb审核，重写mongodb查询等 (#900)

eddba92

LeoQuote mentioned this pull request Nov 2, 2020

升级到v1.8.0后mongodb无法连接实例 #952

Closed

LeoQuote mentioned this pull request May 15, 2024

修复mongodb无法上线 #2649

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

加入mongodb审核，重写mongodb查询等 #900

加入mongodb审核，重写mongodb查询等 #900

fancy-lee commented Sep 30, 2020 •

edited

Loading

LeoQuote left a comment

LeoQuote commented Sep 30, 2020

fancy-lee commented Oct 1, 2020

LeoQuote commented Oct 1, 2020

fancy-lee commented Oct 1, 2020

LeoQuote commented Oct 2, 2020

LeoQuote commented Oct 2, 2020

codecov bot commented Oct 2, 2020 •

edited

Loading

fancy-lee commented Oct 2, 2020

LeoQuote commented Oct 2, 2020

LeoQuote commented Oct 6, 2020

LeoQuote Oct 6, 2020

fancy-lee Oct 6, 2020

LeoQuote Oct 6, 2020

fancy-lee Oct 7, 2020

fancy-lee commented Oct 6, 2020

LeoQuote left a comment •

edited

Loading

LeoQuote Oct 7, 2020

fancy-lee commented Oct 8, 2020

LeoQuote commented Oct 9, 2020

fancy-lee commented Oct 11, 2020

LeoQuote left a comment

hhyo commented Oct 12, 2020

hhyo Oct 16, 2020

hhyo Oct 16, 2020

加入mongodb审核，重写mongodb查询等 #900

加入mongodb审核，重写mongodb查询等 #900

Conversation

fancy-lee commented Sep 30, 2020 • edited Loading

LeoQuote left a comment

Choose a reason for hiding this comment

LeoQuote commented Sep 30, 2020

fancy-lee commented Oct 1, 2020

LeoQuote commented Oct 1, 2020

fancy-lee commented Oct 1, 2020

LeoQuote commented Oct 2, 2020

LeoQuote commented Oct 2, 2020

codecov bot commented Oct 2, 2020 • edited Loading

Codecov Report

fancy-lee commented Oct 2, 2020

LeoQuote commented Oct 2, 2020

LeoQuote commented Oct 6, 2020

LeoQuote Oct 6, 2020

Choose a reason for hiding this comment

fancy-lee Oct 6, 2020

Choose a reason for hiding this comment

LeoQuote Oct 6, 2020

Choose a reason for hiding this comment

fancy-lee Oct 7, 2020

Choose a reason for hiding this comment

fancy-lee commented Oct 6, 2020

LeoQuote left a comment • edited Loading

Choose a reason for hiding this comment

LeoQuote Oct 7, 2020

Choose a reason for hiding this comment

fancy-lee commented Oct 8, 2020

LeoQuote commented Oct 9, 2020

fancy-lee commented Oct 11, 2020

LeoQuote left a comment

Choose a reason for hiding this comment

hhyo commented Oct 12, 2020

hhyo Oct 16, 2020

Choose a reason for hiding this comment

hhyo Oct 16, 2020

Choose a reason for hiding this comment

fancy-lee commented Sep 30, 2020 •

edited

Loading

codecov bot commented Oct 2, 2020 •

edited

Loading

LeoQuote left a comment •

edited

Loading