add benchmark results + issues/140

CHANGELOG.md

@@ -19,6 +19,8 @@ most recent version is listed first.
   - fixed a potential memory leak bug in `naz.correlater.SimpleCorrelater`
   - added a configurable `connection_timeout` which is the duration that `naz` will wait, for connection related activities with SMSC, before timing out.
   - `naz` is now able to re-establish connection and re-bind if the connection between it and SMSC is disconnected.  
+- renamed `connection_timeout` to `socket_timeout`: https://github.com/komuw/naz/pull/141
+- added benchmarks results: https://github.com/komuw/naz/pull/141
 
 
 ## **version:** v0.6.0-beta.1

README.md

@@ -35,7 +35,9 @@ naz is in active development and it's API may change in backward incompatible wa
     + [integration with bug trackers(eg Sentry )](#23-integration-with-bug-trackers)
   + [Rate limiting](#3-rate-limiting)            
   + [Throttle handling](#4-throttle-handling)            
-  + [Queuing](#5-queuing)            
+  + [Queuing](#5-queuing)      
+
+[Benchmarks](./benchmarks/README.md)
 
 
 ## Installation
@@ -65,15 +67,14 @@ cli = naz.Client(
 # queue messages to send
 for i in range(0, 4):
     print("submit_sm round:", i)
-    item_to_enqueue = {
-        "version": "1",
-        "smpp_command": naz.SmppCommand.SUBMIT_SM,
-        "short_message": "Hello World-{0}".format(str(i)),
-        "log_id": "myid12345",
-        "source_addr": "254722111111",
-        "destination_addr": "254722999999",
-    }
-    loop.run_until_complete(outboundqueue.enqueue(item_to_enqueue))
+    loop.run_until_complete(
+        cli.submit_sm(
+            short_message="Hello World-{0}".format(str(i)),
+            log_id="myid12345",
+            source_addr="254722111111",
+            destination_addr="254722999999",
+        )
+    )
 
 # connect to the SMSC host
 reader, writer = loop.run_until_complete(cli.connect())
@@ -374,15 +375,14 @@ then in your application, queue items to the queue;
 ```python
 # queue messages to send
 for i in range(0, 4):
-    item_to_enqueue = {
-        "version": "1",
-        "smpp_command": naz.SmppCommand.SUBMIT_SM,
-        "short_message": "Hello World-{0}".format(str(i)),
-        "log_id": "myid12345",
-        "source_addr": "254722111111",
-        "destination_addr": "254722999999",
-    }
-    loop.run_until_complete(outboundqueue.enqueue(item_to_enqueue))
+    loop.run_until_complete(
+        cli.submit_sm(
+            short_message="Hello World-{0}".format(str(i)),
+            log_id="myid12345",
+            source_addr="254722111111",
+            destination_addr="254722999999",
+        )
+    )
 ```                   
 
 
@@ -442,15 +442,14 @@ then queue on your application side;
 # queue messages to send
 for i in range(0, 5):
     print("submit_sm round:", i)
-    item_to_enqueue = {
-        "version": "1",
-        "smpp_command": naz.SmppCommand.SUBMIT_SM,
-        "short_message": "Hello World-{0}".format(str(i)),
-        "log_id": "myid12345",
-        "source_addr": "254722111111",
-        "destination_addr": "254722999999",
-    }
-    loop.run_until_complete(outboundqueue.enqueue(item_to_enqueue))
+    loop.run_until_complete(
+        cli.submit_sm(
+            short_message="Hello World-{0}".format(str(i)),
+            log_id="myid12345",
+            source_addr="254722111111",
+            destination_addr="254722999999",
+        )
+    )
 ```
 
 

benchmarks/README.md

@@ -1,30 +1,42 @@
+### Benchmarking naz
+This benchmarks were ran using [naz version v0.6.1](https://github.com/komuw/naz/blob/master/CHANGELOG.md#version-v061)      
+They were ran primarily to figure out;
+- how fault tolerant `naz` is
+- whether `naz` has any memory leaks.
 
 ### Methodology
 The benchmark was run this way:
+
 - `naz-cli` was deployed on a $5/month digitalocean server with 1GB of RAM and 1 cpu in the San Francisco region.       
 - A mock SMSC server was deployed on a $5/month digitalocean server with 1GB of RAM and 1 cpu in the Amsterdam region.     
 - A redis server was also deployed to the same $5/month digitalocean server in the Amsterdam region.   
 - The ping latency between the `naz` server in Sanfrancisco and the SMSC server in Amsterdam was about `154 ms`    
 - Approximately 100,000 messages were queued on the redis server.   
 - `naz-cli` would consume the messages from the redis queue and send them out to the SMSC.   
-- In a loop; the `SMSC` would run for a duration of between 13-16 minutes stop for a duration of 1-3 minutes then continue etc.
-- In a loop; the `redis server` would run for a duration of between 13-16 minutes stop for a duration of 1-3 minutes then continue etc.
-- All this while, `naz-cli` is still accessing the redis server and SMSC, re-establishing connections when neccessary.
+- In a loop; the `SMSC` would run for a duration of between 13-16 minutes, stop for a duration of 1-3 minutes then continue etc.
+- In a loop; the `redis server` would run for a duration of between 13-16 minutes, stop for a duration of 1-3 minutes then continue etc.
+- All this while, `naz-cli` is accessing the redis server and SMSC; re-establishing connections when neccessary.
 - All logs from `naz` were been sent to a timescaleDB container for later analysis.   
 - Container/host and custom metrics were also been sent to a prometheus container for later analysis.  
 
 
 ### Results:
-- TODO
+- `naz-cli` used a near constant amount of memory(19MB) throughout the whole benchmarking period. There was no indication of memory leaks.
+![naz-cli memory usage](./static/naz_mem_usage.png "naz-cli memory usage")   
+- `naz-cli` was able to stay up during the whole period. This is despite the fact that the redis server and SMSC servers would 'go down' every 15minutes or so.     
+   `naz-cli` was able to reconnect and re-bind whenever the SMSC came back up again
+- During the running of these benchmarks, a number of bugs were identified in `naz`. Those bugs were fixed in [PR127](https://github.com/komuw/naz/pull/127). Eventually the benchmarks were re-ran with the fixed `naz` version.
+
+
+
 
 
+#### misc.
 connect:   
 ```sh
 psql --host=localhost --port=5432 --username=myuser --dbname=mydb
 ```
 
-
-
 ```sql
 SELECT * FROM logs ORDER BY timestamp DESC LIMIT 5;
 ```
@@ -36,12 +48,6 @@ SELECT * FROM logs WHERE length(logs.error) > 0 ORDER BY timestamp DESC;
 
 ```sql
 SELECT event,log_id,error FROM logs WHERE length(logs.error) > 0 ORDER BY timestamp DESC;
+Copy (SELECT * FROM logs WHERE length(logs.error) > 0 ORDER BY timestamp DESC) To '/tmp/errors.csv' With CSV DELIMITER ',';
+cp /tmp/errors.csv /usr/src/app/
 ```
-
-```sh
-          event          |   log_id    |                error
--------------------------+-------------+--------------------------------------
- naz.Client.send_data    | 405-jctnhvo | [Errno 104] Connection reset by peer
- naz.Client.receive_data |             | [Errno 104] Connection reset by peer
- naz.Client.send_data    | 78-zbtrpxw  | [Errno 104] Connection reset by peer
- ```

benchmarks/app.py

@@ -21,7 +21,7 @@
     outboundqueue=MyRedisQueue(),
     hook=BenchmarksHook(),
     log_handler=BenchmarksLogger("naz.client"),
-    connection_timeout=15.00,
+    socket_timeout=15.00,
     enquire_link_interval=80.00,
     address_range="^{0}".format(
         country_code

benchmarks/digitalocean.sh

@@ -23,7 +23,7 @@ apt -y install docker-ce && \
 usermod -aG docker $(whoami) && \
 pip install -U docker-compose
 
-wget https://github.com/komuw/naz/archive/issues/103.zip && \
+wget https://github.com/komuw/naz/archive/master.zip && \
 unzip master.zip && \
 mv naz-master/ naz && \
 cd naz/benchmarks

documentation/config.md

@@ -60,7 +60,7 @@ hook | python class instance implemeting functionality/hooks to be called by `na
 throttle_handler | python class instance implementing functionality of what todo when naz starts getting throttled responses from SMSC | naz.throttle.SimpleThrottleHandler
 correlation_handler | A python class instance that naz uses to store relations between SMPP sequence numbers and user applications' log_id's and/or hook_metadata. | naz.correlater.SimpleCorrelater
 drain_duration | duration in seconds that `naz` will wait for after receiving a termination signal. | 8.00   
-connection_timeout | duration that `naz` will wait, for connection related activities with SMSC, before timing out | 30.00     
+socket_timeout | duration that `naz` will wait, for socket/connection related activities with SMSC, before timing out | 30.00     
 
 `SMSC`: Short Message Service Centre, ie the server               
 `ESME`: External Short Message Entity, ie the client                   

documentation/sphinx-docs/introduction.rst

@@ -46,15 +46,14 @@ naz is in active development and it's API may change in backward incompatible wa
     # queue messages to send
     for i in range(0, 4):
         print("submit_sm round:", i)
-        item_to_enqueue = {
-            "version": "1",
-            "smpp_command": naz.SmppCommand.SUBMIT_SM,
-            "short_message": "Hello World-{0}".format(str(i)),
-            "log_id": "myid12345",
-            "source_addr": "254722111111",
-            "destination_addr": "254722999999",
-        }
-        loop.run_until_complete(outboundqueue.enqueue(item_to_enqueue))
+        loop.run_until_complete(
+        cli.submit_sm(
+            short_message="Hello World-{0}".format(str(i)),
+            log_id="myid12345",
+            source_addr="254722111111",
+            destination_addr="254722999999",
+        )
+    )
 
     # connect to the SMSC host
     reader, writer = loop.run_until_complete(cli.connect())
@@ -420,28 +419,26 @@ An example of using that queue;
 
     # queue messages to send
     for i in range(0, 4):
-        item_to_enqueue = {
-            "version": "1",
-            "smpp_command": naz.SmppCommand.SUBMIT_SM,
-            "short_message": "Hello World-{0}".format(str(i)),
-            "log_id": "myid12345",
-            "source_addr": "254722111111",
-            "destination_addr": "254722999999",
-        }
-        loop.run_until_complete(outboundqueue.enqueue(item_to_enqueue))
+        loop.run_until_complete(
+        cli.submit_sm(
+            short_message="Hello World-{0}".format(str(i)),
+            log_id="myid12345",
+            source_addr="254722111111",
+            destination_addr="254722999999",
+        )
+    )
 
 then in your application, queue items to the queue;
 
 .. code-block:: python
 
     # queue messages to send
     for i in range(0, 4):
-        item_to_enqueue = {
-            "version": "1",
-            "smpp_command": naz.SmppCommand.SUBMIT_SM,
-            "short_message": "Hello World-{0}".format(str(i)),
-            "log_id": "myid12345",
-            "source_addr": "254722111111",
-            "destination_addr": "254722999999",
-        }
-        loop.run_until_complete(outboundqueue.enqueue(item_to_enqueue))
+        loop.run_until_complete(
+        cli.submit_sm(
+            short_message="Hello World-{0}".format(str(i)),
+            log_id="myid12345",
+            source_addr="254722111111",
+            destination_addr="254722999999",
+        )
+    )