Tigase XMPP Server automated tests reports

This is a page with reports for the automatic tests we run for each public release of the Tigase server. In fact we run these tests much more often. They are part of our development.

This is the new page with redesigned layout for test reports. The old page for all older versions is available here. We are now running more and more tests for each release to limit number of bugs to minimum thus the new page to present results in more convenient way.

What tests do we run? There are 4 major automatic tests scenarios we run:

  1. Functional tests - published for each release for all databases we support: MySQL, PostgreSQL, SQL Server, MongoDB, and Derby. There are also 3 variants for each database the server can be configured in: default authentication, stored procedures based (SP) authentication and authentication based on custom SQL queries (SQL). Functional tests are run for all of them.
  2. Performance tests - published for each major release for all databases we support: MySQL, PostgreSQL, SQL Server, MongoDB, and Derby.
  3. Load tests - we run them nightly on 4 virtual machines. Load Tests
  4. Clustering tests - we run them during our development process to make sure the server works correctly in cluster mode. See the link above for a demonstration.

Functional tests

Basic tests to check all the functionality of the server

Test results
~
(Passed | Failed | Total number of tests | Running time)
Date Version MySQLPostgreSQLDerbyMS SQL MongoDB
2017 Oct 11 7.1.2-b4471 235 | 0 | 235 | 0:3:37:52 235 | 0 | 235 | 0:3:35:118 235 | 0 | 235 | 0:3:39:129 235 | 0 | 235 | 0:3:37:927 235 | 0 | 235 | 0:3:33:831
2017 Aug 30 7.1.1-b4457 235 | 0 | 235 | 0:3:37:235 235 | 0 | 235 | 0:3:49:334 235 | 0 | 235 | 0:3:39:552 235 | 0 | 235 | 0:3:38:644 235 | 0 | 235 | 0:3:34:159
2017 Jan 30 7.1.0-b4379 235 | 0 | 235 | 0:3:36:528 235 | 0 | 235 | 0:3:34:924 235 | 0 | 235 | 0:3:41:464 235 | 0 | 235 | 0:3:52:77 235 | 0 | 235 | 0:3:34:300
2016 Apr 28 7.0.4-b3844 235 | 0 | 235 | 0:3:38:724 235 | 0 | 235 | 0:3:38:600 235 | 0 | 235 | 0:3:40:307 235 | 0 | 235 | 0:3:41:700 235 | 0 | 235 | 0:3:43:47
2016 Mar 9 7.0.3-b3834 235 | 0 | 235 | 0:3:39:183 235 | 0 | 235 | 0:3:35:683 235 | 0 | 235 | 0:3:43:88 235 | 0 | 235 | 0:3:42:297 235 | 0 | 235 | 0:3:40:550
2016 Mar 7 7.0.2-b3821 235 | 0 | 235 | 0:3:39:402 235 | 0 | 235 | 0:3:35:941 235 | 0 | 235 | 0:3:52:84 235 | 0 | 235 | 0:3:49:319 235 | 0 | 235 | 0:3:43:559
2015 Mar 31 7.0.1-b3810 235 | 0 | 235 | 0:3:35:621 235 | 0 | 235 | 0:3:33:203 235 | 0 | 235 | 0:3:39:278 235 | 0 | 235 | 0:3:38:146 235 | 0 | 235 | 0:3:46:246
2015 Feb 23 7.0.0-b3802 235 | 0 | 235 | 0:3:36:575 235 | 0 | 235 | 0:3:33:567 235 | 0 | 235 | 0:3:40:66 235 | 0 | 235 | 0:3:37:855 235 | 0 | 235 | 0:3:32:283
2014 Nov 14 5.2.3-b3470 235 | 0 | 235 | 0:3:32:575 235 | 0 | 235 | 0:3:29:67 235 | 0 | 235 | 0:3:44:114 235 | 0 | 235 | 0:3:32:134 ---
2014 Sep 24 5.2.2-b3463 235 | 0 | 235 | 0:3:30:48 235 | 0 | 235 | 0:3:30:264 235 | 0 | 235 | 0:3:33:603 235 | 0 | 235 | 0:3:44:985 ---
2014 Sep 24 5.2.1-b3461 235 | 0 | 235 | 0:3:29:853 235 | 0 | 235 | 0:3:27:951 235 | 0 | 235 | 0:3:41:358 235 | 0 | 235 | 0:3:33:474 ---
2014 Feb 13 5.2.0-b3447 235 | 0 | 235 | 0:3:36:988 235 | 0 | 235 | 0:3:30:43 235 | 0 | 235 | 0:3:35:387 235 | 0 | 235 | 0:3:40:795 ---
2014 Jan 24 5.2.0-b3437-rc2 235 | 0 | 235 | 0:3:36:507 235 | 0 | 235 | 0:3:31:18 235 | 0 | 235 | 0:3:36:964 235 | 0 | 235 | 0:3:50:689 ---
2013 Aug 19 5.2.0-b3223-beta2 228 | 0 | 228 | 0:3:32:675 228 | 0 | 228 | 0:3:45:879 228 | 0 | 228 | 0:3:41:336 --- ---
2012 Nov 2 5.1.3-b2983 213 | 0 | 213 | 0:3:14:53 213 | 0 | 213 | 0:3:14:542 213 | 0 | 213 | 0:3:14:815 --- ---
2012 Oct 25 5.1.2-b2972 213 | 0 | 213 | 0:3:12:297 213 | 0 | 213 | 0:3:8:244 213 | 0 | 213 | 0:3:17:685 --- ---
2012 Oct 25 5.1.1-b2970 213 | 0 | 213 | 0:3:7:238 213 | 0 | 213 | 0:3:9:706 213 | 0 | 213 | 0:3:15:56 --- ---
2012 Aug 21 5.1.0-b3002 213 | 0 | 213 | 0:2:53:427 213 | 0 | 213 | 0:2:54:79 213 | 0 | 213 | 0:3:19:156 --- ---
2012 Aug 19 5.1.0-b2994 213 | 0 | 213 | 0:2:54:938 213 | 0 | 213 | 0:2:51:954 213 | 0 | 213 | 0:2:56:66 --- ---
2010 Mar 20 5.0.0-b2135 133 | 0 | 133 | 0:1:5:515 133 | 0 | 133 | 0:1:4:25 133 | 0 | 133 | 0:1:8:661 --- ---
2010 Mar 8 5.0.0-b2103 133 | 0 | 133 | 0:1:6:443 133 | 0 | 133 | 0:1:4:844 133 | 0 | 133 | 0:1:17:938 --- ---
2009 Sep 26 4.3.1-b1858 137 | 0 | 137 | 0:1:13:691 137 | 0 | 137 | 0:1:20:294 137 | 0 | 137 | 0:1:27:114 --- ---
2009 Jun 1 4.2.0-b1720 138 | 0 | 138 | 0:0:47:923 138 | 0 | 138 | 0:0:47:994 138 | 0 | 138 | 0:0:50:190 --- ---
2008 Dec 15 4.1.0-b1315 105 | 0 | 105 | 0:0:38:138 105 | 0 | 105 | 0:0:38:850 105 | 0 | 105 | 0:0:43:456 --- ---
2008 Nov 15 4.0.1-b1229 79 | 0 | 79 | 0:0:27:523 79 | 0 | 79 | 0:0:32:569 79 | 0 | 79 | 0:0:38:702 --- ---
2008 Nov 11 4.0.0-b1212 76 | 0 | 76 | 0:0:29:734 76 | 0 | 76 | 0:0:29:783 76 | 0 | 76 | 0:0:34:986 --- ---
2008 Mar 25 3.3.2-b889 36 | 0 | 36 | 0:0:16:171 36 | 0 | 36 | 0:0:16:548 --- --- ---
2008 Mar 15 3.3.2-b880 36 | 0 | 36 | 0:0:14:618 36 | 0 | 36 | 0:0:14:272 --- --- ---
2007 Oct 18 3.0.2-b700 36 | 0 | 36 | 0:0:22:230 36 | 0 | 36 | 0:0:23:231 --- --- ---
2007 Jun 08 2.9.5-b606 36 | 0 | 36 | 0:0:22:810 36 | 0 | 36 | 0:0:23:361 --- --- ---
2007 May 31 2.9.3-b548 36 | 0 | 36 | 0:0:21:850 36 | 0 | 36 | 0:0:24:501 --- --- ---
2007 May 28 2.9.1-b528 36 | 0 | 36 | 0:0:22:240 36 | 0 | 36 | 0:0:22:980 --- --- ---
2007 May 07 2.8.6-b434 36 | 0 | 36 | 0:0:22:570 36 | 0 | 36 | 0:0:23:121 --- --- ---
2007 May 2 2.8.5-b422 36 | 0 | 36 | 0:0:23:130 36 | 0 | 36 | 0:0:22:901 --- --- ---
2007 Apr 26 2.8.3-b409 35 | 0 | 35 | 0:0:27:830 35 | 0 | 35 | 0:0:28:251 --- --- ---
2007 Apr 6 2.7.2-b378 31 | 0 | 31 | 0:0:27:861 31 | 0 | 31 | 0:0:26:710 --- --- ---
2007 Feb 23 2.6.4-b300 31 | 0 | 31 | 0:0:26:841 31 | 0 | 31 | 0:0:28:880 --- --- ---
2007 Feb 15 2.6.4-b295 31 | 0 | 31 | 0:0:26:195 31 | 0 | 31 | 0:0:38:503 --- --- ---
2007 Feb 11 2.6.0-b287 31 | 0 | 31 | 0:0:28:286 31 | 0 | 31 | 0:0:41:28 --- --- ---
2007 Feb 3 2.5.0-b279 31 | 0 | 31 | 0:0:28:156 31 | 0 | 31 | 0:0:39:540 --- --- ---
2007 Jan 27 2.4.0-b263 31 | 0 | 31 | 0:0:27:522 31 | 0 | 31 | 0:0:39:346 --- --- ---

 

Low memory tests

Basic tests to check if server works correctly with max 50MB of RAM
Increased to 75MB on August 21, 2014.

Test results
~
(Passed | Failed | Total number of tests | Running time)
Date Version MySQLPostgreSQLDerbyMS SQL MongoDB
2017 Oct 11 7.1.2-b4471 234 | 0 | 234 | 0:3:32:226 234 | 0 | 234 | 0:3:37:538 234 | 0 | 234 | 0:3:34:516 234 | 0 | 234 | 0:3:33:479 234 | 0 | 234 | 0:3:28:309
2017 Aug 29 7.1.1-b4457 234 | 0 | 234 | 0:3:31:585 234 | 0 | 234 | 0:3:31:701 234 | 0 | 234 | 0:3:35:129 234 | 0 | 234 | 0:3:33:520 234 | 0 | 234 | 0:3:29:692
2017 Jan 30 7.1.0-b4379 234 | 0 | 234 | 0:3:44:630 234 | 0 | 234 | 0:3:30:442 234 | 0 | 234 | 0:3:35:67 234 | 0 | 234 | 0:3:34:751 234 | 0 | 234 | 0:3:28:169
2016 Apr 28 7.0.4-b3844 234 | 0 | 234 | 0:3:34:434 234 | 0 | 234 | 0:3:32:861 234 | 0 | 234 | 0:3:38:542 234 | 0 | 234 | 0:3:36:703 234 | 0 | 234 | 0:3:37:289
2016 Mar 9 7.0.3-b3834 234 | 0 | 234 | 0:3:31:943 234 | 0 | 234 | 0:3:32:840 234 | 0 | 234 | 0:3:37:794 234 | 0 | 234 | 0:3:44:10 234 | 0 | 234 | 0:3:30:625
2016 Mar 7 7.0.2-b3821 234 | 0 | 234 | 0:3:33:157 234 | 0 | 234 | 0:3:32:155 234 | 0 | 234 | 0:3:37:762 234 | 0 | 234 | 0:3:36:911 234 | 0 | 234 | 0:3:29:719
2015 Mar 31 7.0.1-b3810 234 | 0 | 234 | 0:3:30:348 234 | 0 | 234 | 0:3:29:522 234 | 0 | 234 | 0:3:35:742 234 | 0 | 234 | 0:3:48:171 234 | 0 | 234 | 0:3:41:167
2015 Feb 23 7.0.0-b3802 234 | 0 | 234 | 0:3:31:369 234 | 0 | 234 | 0:3:29:109 234 | 0 | 234 | 0:3:50:12 234 | 0 | 234 | 0:3:32:998 234 | 0 | 234 | 0:3:27:784
2014 Nov 14 5.2.3-b3470 234 | 0 | 234 | 0:3:26:546 234 | 0 | 234 | 0:3:25:269 234 | 0 | 234 | 0:3:31:910 234 | 0 | 234 | 0:3:28:345 ---
2014 Sep 24 5.2.2-b3463 234 | 0 | 234 | 0:3:26:407 234 | 0 | 234 | 0:3:23:438 234 | 0 | 234 | 0:3:29:465 234 | 0 | 234 | 0:3:40:298 ---
2014 Sep 24 5.2.1-b3461 234 | 0 | 234 | 0:3:25:498 234 | 0 | 234 | 0:3:24:684 234 | 0 | 234 | 0:3:41:236 234 | 0 | 234 | 0:3:59:295 ---
2014 Feb 13 5.2.0-b3447 234 | 0 | 234 | 0:3:42:783 234 | 0 | 234 | 0:3:26:591 234 | 0 | 234 | 0:3:33:563 234 | 0 | 234 | 0:3:37:183 ---
2014 Jan 24 5.2.0-b3437-rc2 234 | 0 | 234 | 0:3:32:300 234 | 0 | 234 | 0:3:27:487 234 | 0 | 234 | 0:3:34:472 234 | 0 | 234 | 0:3:37:281 ---
2012 Nov 3 5.1.3-b2983 212 | 0 | 212 | 0:3:1:963 212 | 0 | 212 | 0:3:3:239 212 | 0 | 212 | 0:3:6:373 --- ---
2012 Oct 25 5.1.2-b2972 212 | 0 | 212 | 0:3:5:928 212 | 0 | 212 | 0:3:10:993 212 | 0 | 212 | 0:3:4:936 --- ---
2012 Oct 25 5.1.1-b2970 212 | 0 | 212 | 0:3:4:437 212 | 0 | 212 | 0:3:1:486 212 | 0 | 212 | 0:3:8:297 --- ---
2012 Aug 21 5.1.0-b3002 212 | 0 | 212 | 0:2:49:748 212 | 0 | 212 | 0:2:50:869 212 | 0 | 212 | 0:2:52:221 --- ---
2012 Aug 19 5.1.0-b2994 212 | 0 | 212 | 0:2:50:48 212 | 0 | 212 | 0:3:3:434 212 | 0 | 212 | 0:2:51:807 --- ---
2008 Dec 2 4.0.2-b1256 83 | 0 | 83 | 0:0:17:438 83 | 0 | 83 | 0:0:17:305 83 | 0 | 83 | 0:0:36:201 --- ---