Forums before death by AOL, social media and spammers... "We can't have nice things"
| comp.unix.bsd.freebsd.announce | FreeBSD announcements | 143 messages |
[ << oldest | < older | list | newer > | newest >> ]
| Message 134 of 143 |
| Lorenzo Salvadore to All |
| FreeBSD Status Report - Third Quarter 20 |
| 30 Nov 25 14:00:08 |
[continued from previous message]
In this quarter, as part of the infrastructure modernization work commissioned
by the Sovereign Tech Agency (STA), I have been working on the in-tree CI
Makefile targets. I also worked on bringing our CI test reports to a clean
state on our tier-1 architectures (amd64 and aarch64). This report is a
supplement to the overall STA status report and will describe the work done in
more detail.
tests/ci improvements
The tests/ci subdirectory in the src tree was introduced in commit: Add
preliminary in-tree CI infrastructure for developers by Moin Rahman and aims to
provide an easy way for developers to replicate the CI testing run by our
Jenkins cluster. In this quarter, the following improvements were made by the
team:
New functionality:
• tests/ci: run ci-full kyua tests in parallel
• tests/ci: Add KYUA_TEST_FILTERS to allow user to select specific tests
• tests/ci: Add CIENV variable
Bug fixes:
• tests/ci: Use suitable variable for qemu-user-static existence check
• tests/ci: fix race condition in bhyveload boot
• tests/ci: fix missing /usr/local/{sbin,bin} in freebsdci rc PATH
• tests/ci: Fix missing qemu devices
• tests/ci: Fix race condition with ci-extractmeta
• tests/ci: Fix wrong chflags target path in 'beforeclean' target
• tests/ci: Use QEMU blockdev declaration for all platforms
• tests/ci: Fix unescaped kld_list var in rc.conf
With these changes, a developer can run CI with these example commands as root:
# Fully parallel CI:
make ci
# Single-threaded CI
make PARALLEL_JOBS=1 ci
# Single-threaded CI, running a subset of the tests as described in
kyua-test(1)
make PARALLEL_JOBS=1 KYUA_TEST_FILTERS='/path/to/testcase /path/
o/another:testname1' ci
# Run smoke (boot) tests
make CITYPE=smoke ci
Test case management
Tinderbox has been reporting that our supported platforms are failing in CI
since a run from the last quarter. As the backlog grows larger, it becomes
harder for users and developers to notice a new failure and pin it to a
particular commit.
To complement the tests/ci CI/CD automation improvements, along with Bricoler
to help with more granular investigations, I worked on cleaning up the failing
test backlog on tier-1 architectures. The following patches and bug reports
were submitted as a result of this (still ongoing) work:
New bug reports filed to track failing or flaky tests:
• PR 288991: sys/netinet/output:output_raw_flowid_mpath_success
• PR 289096: lib/libexecinfo/sigtramp_test:test_backtrace_sigtramp
• PR 289165: usr.bin/limits/limits_test:cputime_soft_flag
• PR 289240: sys/netlink/netlink_socket:overflow
• PR 289239: sys/netpfil/pf/sctp:pfsync
• PR 289236: sys/kern/exterr_test:gettext_extended
• PR 289382: sys/netinet6/lpm6:lpm6_test1_success
• PR 289628: sys/netpfil/pf/nat:endpoint_independent_pass
• PR 289630: libexec/rc/rc_subr_test:wait_for_pids_progress
• PR 289237: sys/fs/fusefs/last_local_modify:main
• PR 289084: lib/libc/string/memcmp_test:{diff,neq}
• PR 289477: sys/netpfil/pf/
route_to:prefer_ipv6_nexthop_mixed_af_random_table_ipv4
• PR 289299: sys/netpfil/pf/rules_counter:keepcounters
• PR 289684: sys/netlink/test_snl:snl_parse_errmsg_capped
• PR 289146: sbin/ipfw/test_add_rule.py:TestAddRule::test_add_action
Unskip tests that are wrongly skipped in CI:
• tests/sys/netpfil: unskip tests that no longer need to be skipped
• tests/ci: Add missing kmods and pkgs to unskip tests
Test case metadata fixes:
• fix parallel execution of swapon tests
• cap_dns/tests/dns_test: mark tests as needing network access
• pfctl tests: use require.kmods instead of manual check for pf
• cap_net/net_test: require 'allow_network_access'
• tests/mac_portacl: enable is_exclusive for now
• tests/sys/mqueue: use require.kmods property instead of ad-hoc checks
• tests/sys/netlink: use require.kmods property instead of ad-hoc checks
• tests/sys/opencrypto: use require.kmods property instead of ad-hoc checks
• tests/sys/aio: use require.kmods property instead of ad-hoc checks
• tests/pf/ioctl: use require.kmods property instead of ad-hoc checks
• tests/sys/netmap: use require.kmods property instead of ad-hoc checks
• tests/sndstat: use require.kmods property instead of ad-hoc checks
• tests/socket_accf: use require.kmods property instead of ad-hoc checks
• tests/sys/net: use require.kmods property instead of ad-hoc checks
• tests/sys/netinet: use require.kmods property instead of ad-hoc checks
• tests/vmm_cred_jail: use require.kmods property instead of ad-hoc checks
mark tests as "expected fail" (xfail), currently WIP:
• atf_pytest: fix xfail detection from pytest report
Tooling (WIP)
To catch errors more quickly, instead of relying on Jenkins to update the test
report, I ran local CI multiple times daily. To help with this, I worked on
some tooling to speed up the testing/debugging cycles. I am maintaining the
following (currently uncommitted) tools:
• parallel CI runner built on top of tests/ci
• (VERY WIP) automated CI bug report/triage system
Sponsor: The FreeBSD Foundation
━━━━━━━━━━━━━━━━━━━━━
━━━━━━━━━━━━━━━━━━━━━
━━━━━━━━━━━━━━━━━━━━
━━━━━━━━━━━━━━
Support for installing pkgbase systems
Contact: Isaac Freund
|
(c) 1994, bbs@darkrealms.ca