Tigase XMPP Server automated tests reports

This is a page with reports for the automatic tests we run for each public release of the Tigase server. In fact we run these tests much more often. They are part of our development.

This is the new page with redesigned layout for test reports. The old page for all older versions is available here. We are now running more and more tests for each release to limit number of bugs to minimum thus the new page to present results in more convenient way.

What tests do we run? There are 4 major automatic tests scenarios we run:

  1. Functional tests - published for each release for all databases we support: MySQL, PostgreSQL and Derby. There are also 3 variants for each database the server can be configured in: default authentication, stored procedures based (SP) authentication and authentication based on custom SQL queries (SQL). Functional tests are run for all of them.
  2. Performance tests - published for each major release for all databases we support: MySQL, PostgreSQL and Derby.
  3. Load tests - we run them occasionally when we have an access to proper equipment as we don't own hardware good enough to run load tests. The last load tests results are published on a separate page. Click on the 'Load tests' link in the menu on the left hand side of the page.
  4. Clustering tests - we run them during our development process to make sure the server works correctly in the cluster mode. The results are not directly related to any public release yet and they are available on a separate page. To find the report please click 'Clustering tests' link in the menu on the left hand side of the page.

Functional tests

Basic tests to check all the functionality of the server

Test results
~
(Passed | Failed | Total number of tests | Running time)
Date Version MySQLPostgreSQLDerbyMS SQL MongoDB
8.0.0-b10083 220 | 0 | 220 | 0:4:19:556 220 | 0 | 220 | 0:4:15:490 220 | 0 | 220 | 0:4:31:239 --- 220 | 0 | 220 | 0:4:35:885
8.0.0-b09961-RC1 220 | 0 | 220 | 0:4:20:412 220 | 0 | 220 | 0:4:12:58 220 | 0 | 220 | 0:4:15:7 --- 220 | 0 | 220 | 0:4:19:580
7.1.5-b4512 235 | 0 | 235 | 0:3:32:536 235 | 0 | 235 | 0:3:32:418 235 | 0 | 235 | 0:3:31:873 --- 235 | 0 | 235 | 0:3:41:919
7.1.4-b4502 235 | 0 | 235 | 0:3:32:168 235 | 0 | 235 | 0:3:33:835 235 | 0 | 235 | 0:3:31:245 --- 235 | 0 | 235 | 0:3:35:229
7.1.3-b4482 235 | 0 | 235 | 0:3:32:376 235 | 0 | 235 | 0:3:30:322 235 | 0 | 235 | 0:3:29:955 --- 235 | 0 | 235 | 0:3:29:93
7.1.2-b4471 235 | 0 | 235 | 0:3:37:52 235 | 0 | 235 | 0:3:35:118 235 | 0 | 235 | 0:3:39:129 235 | 0 | 235 | 0:3:37:927 235 | 0 | 235 | 0:3:33:831
7.1.1-b4457 235 | 0 | 235 | 0:3:37:235 235 | 0 | 235 | 0:3:49:334 235 | 0 | 235 | 0:3:39:552 235 | 0 | 235 | 0:3:38:644 235 | 0 | 235 | 0:3:34:159
7.1.0-b4379 235 | 0 | 235 | 0:3:36:528 235 | 0 | 235 | 0:3:34:924 235 | 0 | 235 | 0:3:41:464 235 | 0 | 235 | 0:3:52:77 235 | 0 | 235 | 0:3:34:300
7.0.4-b3844 235 | 0 | 235 | 0:3:38:724 235 | 0 | 235 | 0:3:38:600 235 | 0 | 235 | 0:3:40:307 235 | 0 | 235 | 0:3:41:700 235 | 0 | 235 | 0:3:43:47
7.0.3-b3834 235 | 0 | 235 | 0:3:39:183 235 | 0 | 235 | 0:3:35:683 235 | 0 | 235 | 0:3:43:88 235 | 0 | 235 | 0:3:42:297 235 | 0 | 235 | 0:3:40:550
7.0.2-b3821 235 | 0 | 235 | 0:3:39:402 235 | 0 | 235 | 0:3:35:941 235 | 0 | 235 | 0:3:52:84 235 | 0 | 235 | 0:3:49:319 235 | 0 | 235 | 0:3:43:559
7.0.1-b3810 235 | 0 | 235 | 0:3:35:621 235 | 0 | 235 | 0:3:33:203 235 | 0 | 235 | 0:3:39:278 235 | 0 | 235 | 0:3:38:146 235 | 0 | 235 | 0:3:46:246
7.0.0-b3802 235 | 0 | 235 | 0:3:36:575 235 | 0 | 235 | 0:3:33:567 235 | 0 | 235 | 0:3:40:66 235 | 0 | 235 | 0:3:37:855 235 | 0 | 235 | 0:3:32:283
5.2.3-b3470 235 | 0 | 235 | 0:3:32:575 235 | 0 | 235 | 0:3:29:67 235 | 0 | 235 | 0:3:44:114 235 | 0 | 235 | 0:3:32:134 ---
5.2.2-b3463 235 | 0 | 235 | 0:3:30:48 235 | 0 | 235 | 0:3:30:264 235 | 0 | 235 | 0:3:33:603 235 | 0 | 235 | 0:3:44:985 ---
5.2.1-b3461 235 | 0 | 235 | 0:3:29:853 235 | 0 | 235 | 0:3:27:951 235 | 0 | 235 | 0:3:41:358 235 | 0 | 235 | 0:3:33:474 ---
5.2.0-b3447 235 | 0 | 235 | 0:3:36:988 235 | 0 | 235 | 0:3:30:43 235 | 0 | 235 | 0:3:35:387 235 | 0 | 235 | 0:3:40:795 ---
5.2.0-b3437-rc2 235 | 0 | 235 | 0:3:36:507 235 | 0 | 235 | 0:3:31:18 235 | 0 | 235 | 0:3:36:964 235 | 0 | 235 | 0:3:50:689 ---
5.2.0-b3223-beta2 228 | 0 | 228 | 0:3:32:675 228 | 0 | 228 | 0:3:45:879 228 | 0 | 228 | 0:3:41:336 --- ---
5.1.3-b2983 213 | 0 | 213 | 0:3:14:53 213 | 0 | 213 | 0:3:14:542 213 | 0 | 213 | 0:3:14:815 --- ---
5.1.2-b2972 213 | 0 | 213 | 0:3:12:297 213 | 0 | 213 | 0:3:8:244 213 | 0 | 213 | 0:3:17:685 --- ---
5.1.1-b2970 213 | 0 | 213 | 0:3:7:238 213 | 0 | 213 | 0:3:9:706 213 | 0 | 213 | 0:3:15:56 --- ---
5.1.0-b3002 213 | 0 | 213 | 0:2:53:427 213 | 0 | 213 | 0:2:54:79 213 | 0 | 213 | 0:3:19:156 --- ---
5.1.0-b2994 213 | 0 | 213 | 0:2:54:938 213 | 0 | 213 | 0:2:51:954 213 | 0 | 213 | 0:2:56:66 --- ---
5.0.0-b2135 133 | 0 | 133 | 0:1:5:515 133 | 0 | 133 | 0:1:4:25 133 | 0 | 133 | 0:1:8:661 --- ---
5.0.0-b2103 133 | 0 | 133 | 0:1:6:443 133 | 0 | 133 | 0:1:4:844 133 | 0 | 133 | 0:1:17:938 --- ---
4.3.1-b1858 137 | 0 | 137 | 0:1:13:691 137 | 0 | 137 | 0:1:20:294 137 | 0 | 137 | 0:1:27:114 --- ---
4.2.0-b1720 138 | 0 | 138 | 0:0:47:923 138 | 0 | 138 | 0:0:47:994 138 | 0 | 138 | 0:0:50:190 --- ---
4.1.0-b1315 105 | 0 | 105 | 0:0:38:138 105 | 0 | 105 | 0:0:38:850 105 | 0 | 105 | 0:0:43:456 --- ---
4.0.1-b1229 79 | 0 | 79 | 0:0:27:523 79 | 0 | 79 | 0:0:32:569 79 | 0 | 79 | 0:0:38:702 --- ---
4.0.0-b1212 76 | 0 | 76 | 0:0:29:734 76 | 0 | 76 | 0:0:29:783 76 | 0 | 76 | 0:0:34:986 --- ---
3.3.2-b889 36 | 0 | 36 | 0:0:16:171 36 | 0 | 36 | 0:0:16:548 --- --- ---
3.3.2-b880 36 | 0 | 36 | 0:0:14:618 36 | 0 | 36 | 0:0:14:272 --- --- ---
3.0.2-b700 36 | 0 | 36 | 0:0:22:230 36 | 0 | 36 | 0:0:23:231 --- --- ---
2.9.5-b606 36 | 0 | 36 | 0:0:22:810 36 | 0 | 36 | 0:0:23:361 --- --- ---
2.9.3-b548 36 | 0 | 36 | 0:0:21:850 36 | 0 | 36 | 0:0:24:501 --- --- ---
2.9.1-b528 36 | 0 | 36 | 0:0:22:240 36 | 0 | 36 | 0:0:22:980 --- --- ---
2.8.6-b434 36 | 0 | 36 | 0:0:22:570 36 | 0 | 36 | 0:0:23:121 --- --- ---
2.8.5-b422 36 | 0 | 36 | 0:0:23:130 36 | 0 | 36 | 0:0:22:901 --- --- ---
2.8.3-b409 35 | 0 | 35 | 0:0:27:830 35 | 0 | 35 | 0:0:28:251 --- --- ---
2.7.2-b378 31 | 0 | 31 | 0:0:27:861 31 | 0 | 31 | 0:0:26:710 --- --- ---
2.6.4-b300 31 | 0 | 31 | 0:0:26:841 31 | 0 | 31 | 0:0:28:880 --- --- ---
2.6.4-b295 31 | 0 | 31 | 0:0:26:195 31 | 0 | 31 | 0:0:38:503 --- --- ---
2.6.0-b287 31 | 0 | 31 | 0:0:28:286 31 | 0 | 31 | 0:0:41:28 --- --- ---
2.5.0-b279 31 | 0 | 31 | 0:0:28:156 31 | 0 | 31 | 0:0:39:540 --- --- ---
2.4.0-b263 31 | 0 | 31 | 0:0:27:522 31 | 0 | 31 | 0:0:39:346 --- --- ---

 

Low memory tests

Basic tests to check if server works correctly with max 50MB of RAM
Increased to 75MB on August 21, 2014.

Test results
~
(Passed | Failed | Total number of tests | Running time)
Date Version MySQLPostgreSQLDerbyMS SQL MongoDB
8.0.0-b10083 219 | 0 | 219 | 0:3:34:622 219 | 0 | 219 | 0:3:29:900 219 | 3 | 222 | 0:6:15:537 --- 219 | 0 | 219 | 0:3:33:555
8.0.0-b09961-RC1 219 | 0 | 219 | 0:3:33:637 219 | 0 | 219 | 0:3:35:426 219 | 0 | 219 | 0:3:38:36 --- 219 | 0 | 219 | 0:3:37:635
7.1.5-b4512 234 | 0 | 234 | 0:3:23:399 234 | 0 | 234 | 0:3:25:239 234 | 0 | 234 | 0:3:22:731 --- 234 | 0 | 234 | 0:3:18:906
7.1.4-b4502 234 | 0 | 234 | 0:3:28:84 234 | 0 | 234 | 0:3:20:612 234 | 0 | 234 | 0:3:22:999 --- 234 | 0 | 234 | 0:3:17:786
7.1.3-b4482 234 | 0 | 234 | 0:3:28:60 234 | 0 | 234 | 0:3:20:459 234 | 0 | 234 | 0:3:21:806 --- 234 | 0 | 234 | 0:3:18:514
7.1.2-b4471 234 | 0 | 234 | 0:3:32:226 234 | 0 | 234 | 0:3:37:538 234 | 0 | 234 | 0:3:34:516 234 | 0 | 234 | 0:3:33:479 234 | 0 | 234 | 0:3:28:309
7.1.1-b4457 234 | 0 | 234 | 0:3:31:585 234 | 0 | 234 | 0:3:31:701 234 | 0 | 234 | 0:3:35:129 234 | 0 | 234 | 0:3:33:520 234 | 0 | 234 | 0:3:29:692
7.1.0-b4379 234 | 0 | 234 | 0:3:44:630 234 | 0 | 234 | 0:3:30:442 234 | 0 | 234 | 0:3:35:67 234 | 0 | 234 | 0:3:34:751 234 | 0 | 234 | 0:3:28:169
7.0.4-b3844 234 | 0 | 234 | 0:3:34:434 234 | 0 | 234 | 0:3:32:861 234 | 0 | 234 | 0:3:38:542 234 | 0 | 234 | 0:3:36:703 234 | 0 | 234 | 0:3:37:289
7.0.3-b3834 234 | 0 | 234 | 0:3:31:943 234 | 0 | 234 | 0:3:32:840 234 | 0 | 234 | 0:3:37:794 234 | 0 | 234 | 0:3:44:10 234 | 0 | 234 | 0:3:30:625
7.0.2-b3821 234 | 0 | 234 | 0:3:33:157 234 | 0 | 234 | 0:3:32:155 234 | 0 | 234 | 0:3:37:762 234 | 0 | 234 | 0:3:36:911 234 | 0 | 234 | 0:3:29:719
7.0.1-b3810 234 | 0 | 234 | 0:3:30:348 234 | 0 | 234 | 0:3:29:522 234 | 0 | 234 | 0:3:35:742 234 | 0 | 234 | 0:3:48:171 234 | 0 | 234 | 0:3:41:167
7.0.0-b3802 234 | 0 | 234 | 0:3:31:369 234 | 0 | 234 | 0:3:29:109 234 | 0 | 234 | 0:3:50:12 234 | 0 | 234 | 0:3:32:998 234 | 0 | 234 | 0:3:27:784
5.2.3-b3470 234 | 0 | 234 | 0:3:26:546 234 | 0 | 234 | 0:3:25:269 234 | 0 | 234 | 0:3:31:910 234 | 0 | 234 | 0:3:28:345 ---
5.2.2-b3463 234 | 0 | 234 | 0:3:26:407 234 | 0 | 234 | 0:3:23:438 234 | 0 | 234 | 0:3:29:465 234 | 0 | 234 | 0:3:40:298 ---
5.2.1-b3461 234 | 0 | 234 | 0:3:25:498 234 | 0 | 234 | 0:3:24:684 234 | 0 | 234 | 0:3:41:236 234 | 0 | 234 | 0:3:59:295 ---
5.2.0-b3447 234 | 0 | 234 | 0:3:42:783 234 | 0 | 234 | 0:3:26:591 234 | 0 | 234 | 0:3:33:563 234 | 0 | 234 | 0:3:37:183 ---
5.2.0-b3437-rc2 234 | 0 | 234 | 0:3:32:300 234 | 0 | 234 | 0:3:27:487 234 | 0 | 234 | 0:3:34:472 234 | 0 | 234 | 0:3:37:281 ---
5.1.3-b2983 212 | 0 | 212 | 0:3:1:963 212 | 0 | 212 | 0:3:3:239 212 | 0 | 212 | 0:3:6:373 --- ---
5.1.2-b2972 212 | 0 | 212 | 0:3:5:928 212 | 0 | 212 | 0:3:10:993 212 | 0 | 212 | 0:3:4:936 --- ---
5.1.1-b2970 212 | 0 | 212 | 0:3:4:437 212 | 0 | 212 | 0:3:1:486 212 | 0 | 212 | 0:3:8:297 --- ---
5.1.0-b3002 212 | 0 | 212 | 0:2:49:748 212 | 0 | 212 | 0:2:50:869 212 | 0 | 212 | 0:2:52:221 --- ---
5.1.0-b2994 212 | 0 | 212 | 0:2:50:48 212 | 0 | 212 | 0:3:3:434 212 | 0 | 212 | 0:2:51:807 --- ---
4.0.2-b1256 83 | 0 | 83 | 0:0:17:438 83 | 0 | 83 | 0:0:17:305 83 | 0 | 83 | 0:0:36:201 --- ---