This repo holds patterns for finding non-determinism and missed error handling in C, Python and Go source code, and rules for Lua source code.
The rules currently supports semgrep and coccinelle.
To run a single semgrep rule:
$ semgrep -f rules/<lang>/<rule>.yml .
To run all semgrep rules:
$ semgrep --config rules/<lang>/
To run a single coccinelle rule:
$ spatch --sp-file coccinelle/mmap_map_failed.cocci --very-quiet --dir ~/sources/tarantool/src/
| ID | Description |
|---|---|
| lua_builtin_global_write | Global write. |
| lua_cmp_by_reference | Compare uncomparable. |
| lua_cmp_float_numbers | Compare float numbers. |
| lua_comparison_precedence | Comparison precedence. |
| lua_debug_enabled | Using debug.sethook(). |
| lua_deprecated_api | Using deprecated Lua function(s). |
| lua_deprecated_global | Using deprecated globals. |
| lua_duplicate_condition | Duplicate conditions. |
| lua_duplicate_function | Duplicate functions. |
| lua_unsafe_function | Duplicate locals. |
| lua_fd_leak | File descriptors leaks. |
| lua_for_range | Incorrect for bounds. |
| lua_format_string | Mistakes in a format string. |
| lua_func_inside_func | A function is defined inside another function. |
| lua_function_unused | Unused function. |
| lua_global_used_as_local | Global is used as a local. |
| lua_implicit_return | Implicit return. |
| lua_import_unused | Unused import. |
| lua_init_rng_without_seed | math.random() is used without random seed. |
| lua_integer_parsing | Incorrect using integers. |
| lua_local_shadow | Using local shadow. |
| lua_local_unused | Unused local. |
| lua_magic_number | Using magic numbers. |
| lua_misleading_and_or | Misleading and and or. |
| lua_multi_line_statement | Multiline statement. |
| lua_pcall_err_handling | Ignoring pcall results handling. |
| lua_pcall_with_method | Use pcall(self.func, self). |
| lua_placeholder_write | Placeholder read. |
| lua_print_unprintable | Print unprintable. |
| lua_require_inside_func | Import is used inside a function. |
| lua_same_line_statement | Same line statement. |
| lua_setting_looping_variables | Mutate immutable variables. |
| lua_table_first_index | Using index 0 in tables. |
| lua_table_len | Using # for a table length. |
| lua_table_literal | Table literal. |
| lua_table_operations | Table operations. |
| lua_unbalanced_assignment | Unbalanced assignment. |
| lua_uninitialized_local | Non-initialized local. |
| lua_unknown_global | Using unknown global variable. |
| lua_unknown_type | Unknown type. |
| lua_unreachable_code | Unreachable code. |
| lua_unsafe_function | Unsafe function. |
| lua_use_fd_after_close | File descriptors leak. |
| lua_writing_to_file_in_read_mode | Writing to a file opened in read-only mode. |
| luajit_2.1 | Function is compiled starting with LuaJIT 2.1. |
| luajit_jit_off | JIT compilation is turned off. |
| luajit_nyi | Function is not JIT-compiled. |
| luajit_partial_compilation | Function may be JIT-compiled, depending on the circumstances. |
| tarantool_box_cfg_raw_access | Raw access to a box.cfg table. |
| tarantool_box_grant_guest_full_access | Full access for a guest user to universe. |
| tarantool_box_insert_nil | Insert a nil to a space. |
| tarantool_box_missed_if_not_exist | Option if_not_exists is missed. |
| tarantool_box_once | Using box.once(). |
| tarantool_box_select_nil | Using nil in :select(). |
| tarantool_box_set_trigger_once | Set trigger only once. |
| tarantool_box_space_format | |
| tarantool_box_space_name | Rules for object names. |
| tarantool_crypto_insecure_hash_algorithm | Using insecure hash algorithm. |
| tarantool_deprecated_c_api | Using deprecated C/C++ function(s). |
| tarantool_deprecated_lua_api | Using deprecated Tarantool Lua function(s). |
| tarantool_digest_insecure_hash_algorithm | Using insecure hash algorithm. |
| tarantool_digest_insecure_hash_algorithm | |
| tarantool_insecure-hash-algorithm-sha1 | |
| tarantool_fiber_missed_name | Fiber name is not set. |
| tarantool_fiber_missed_testcancel | Fiber method :testcancel() is missed. |
| tarantool_fiber_missed_timeout | Timeout is missed in a fiber method. |
| tarantool_fiber_missed_yield | Fiber yield is missed. |
| tarantool_fio_bad_file_permission | Bad file permissions. |
| tarantool_fio_fd_leak | File descriptor leaks. |
| tarantool_fio_use_fd_after_close | Using file descriptor after close. |
| tarantool_fio_writing_to_file_in_read_mode | Writing to a file opened with read-only mode. |
| tarantool_http_client_no_cert_validation | Certificate validation is disabled. |
| tarantool_http_client_response_json | Method :json() can raise an error. |
| tarantool_luatest_missed_test_prefix | A test prefix is missed a test title. |
| tarantool_net_box_fd_leak | File descriptor leaks. |
| tarantool_net_box_missed_timeout | Timeout is not set. |
| tarantool_popen_enabled_shell | Shell is enabled in popen. |
| tarantool_popen_fd_leak | File descriptor leaks. |
| tarantool_popen_use_fd_after_close | Using file descriptor after close. |
| tarantool_socket_bind_to_all_interfaces | Socket bind to 0.0.0.0 or ::. |
| tarantool_socket_fd_leak | File descriptor leaks. |
| tarantool_socket_fd_leak | |
| tarantool_socket_missed_timeout | Timeout is not set. |
| tarantool_socket_use_fd_after_close | Using socket handler after close. |
| tarantool_vshard_bad_hash_func | Sharding hash algorithm is bad. |
| tarantool_vshard_no_timeouts | Timeout is not set. |
Non-determinism is a source of test flakiness.
References
- An empirical analysis of flaky tests - Qingzhou Luo, Farah Hariri, Lamyaa Eloussi, Darko Marinov
- Empirical Analysis of Factors and their Effect on Test Flakiness - Practitioners’ Perceptions - Azeem Ahmad, Ola Leifler, Kristian Sandahl
- Root Causing Flaky Tests in a Large-Scale Industrial Setting - Wing Lam, Patrice Godefroid, Suman Nath, Anirudh Santhiar, Suresh Thummalapenta
- What is the Vocabulary of Flaky Tests? - Gustavo Pinto, Breno Miranda, Supun Dissanayake, Marcelo d'Amorim, Christoph Treude, Antonia Bertolino
- Eradicating Non-Determinism in Tests - Martin Fowler
Error handling is importance for application reliability. Nice description was given by Dan Luu:
Proper error handling code is hard. Bugs in error handling code are a major cause of bad problems. This means that the probability of having sequential bugs, where an error causes buggy error handling code to run, isn't just the independent probabilities of the individual errors multiplied. It's common to have cascading failures cause a serious outage. There's a sense in which this is obvious -- error handling is generally regarded as being hard. If I mention this to people they'll tell me how obvious it is that a disproportionate number of serious postmortems come out of bad error handling and cascading failures where errors are repeatedly not handled correctly. But despite this being “obvious”, it's not so obvious that sufficient test and static analysis effort are devoted to making sure that error handling works.
See also Aspirator, a simple tool that finds serious bugs in Java exception handler.
References
- The Do's and Don'ts of Error Handling - Joe Armstrong (GOTO 2018)
- Finding Error-Handling Bugs in Systems Code Using Static Analysis - Cindy Rubio-González, Ben Liblit
- Simple Testing Can Prevent Most Critical Failures: An Analysis of Production Failures in Distributed Data-Intensive Systems - Ding Yuan, Yu Luo, Xin Zhuang, Guilherme Renna Rodrigues, Xu Zhao, Yongle Zhang, Pranay U. Jain, and Michael Stumm, University of Toronto (USENIX)
- Improving the Quality of Error-Handling Code in Systems Software using Function-Local Information - Suman Saha