Revert "perf(@angular/build): skip semantic affected-file walk when t… #3973

Job	Run time
list	18s
workflow (devkit-core-tests-rerun)	1m 57s
workflow (build-cli)	2m 6s
workflow (devkit-core-tests)	1m 39s
workflow (modules-builder-tests)	2m 18s
workflow (build-unit-tests)	2m 39s
	10m 57s

Provide feedback