Calls to this method may dominate generation time in some cases.
Measurements for configuring cmake itself show a cache hit rate of ~57%
(7753 total calls, 4453 cache hits). For a larger project (that also
makes use of custom targets as prerequisite for all compile targets),
the measured cache hit ratio is ~96% (2530827 total calls, 2433124 cache
hits). For this project the observable cmake runtime could be reduced
from 40s to 30s.
Signed-off-by: Matthias Maennich <matthias@maennich.net>
The commit v3.7.0-rc1~339^2 (Ninja: Fix inter-target order-only
dependencies of custom command, 2016-07-20) might cause performance
degradations for larger projects. Especially when using custom
commands as an input for each compilation rule (e.g. generated headers).
For reference in the following I am referring to
Source/cmGlobalNinjaGenerator.cxx:
-> cmGlobalNinjaGenerator::AppendTargetDependsClosure
-> cmGlobalNinjaGenerator::ComputeTargetDependsClosure
It turned out that the mentioned commit is doing (indirectly) some
redundant work that might impact performance when generating large
projects.
Imagine the dependency tree of custom targets:
A
\
C - D - E
/
B
For each target the transitive closure is calculated recursively, but as
the TargetDependsClosures are only cached on the top most level, everything
downstream has to be recalculated. I.e.
A->C->D->E
B->C->D->E
This ultimately leads to a lot of redundant calls to AppendTargetOutputs.
The recursive nature of the algorithm itself is not significant to the
problem, but reducing the work to actually to be done work, eliminates the
performance problem.
This patch changes the way, intermediate results are cached. Rather than
caching the closure of targets, we cache the closure of outputs. Such that
in the example above at B->C the cache already would kick in.
Caching the outputs has one disadvantage that the patch takes care of.
In case of such a structure
A E
\ / \
C - D G
/ \ /
B F
the calling order for A would be
A->C->D->E->G (at which time G is seen to the recursion)
then the recursion returns until it reaches
A->C->D->F (at which the seen G would prevent to recurse down to G)
But this would poison the cache for F with a wrong value (without G).
Hence we use a local result set to ensure the cache is still consistently
populated.
For a large C++ project with around 25k targets this reduced the CMake
configure / generate time from ~40s to ~29s.
Signed-off-by: Matthias Maennich <matthias@maennich.net>
For CMP0058 we identify dependencies that have no rules to generate them
by collecting a set of all dependencies and a set of all files CMake
knows are generated by something, and then computing a set difference.
Fix construction of these sets to use the same form for each path.
This is after ConvertToNinjaPath but before EncodePath.
Without this, the slash conversion done by EncodePath for the GNU
compiler on Windows results in a mismatch, triggering an incorrect
CMP0058 warning.
This fixes a false-positive `CMP0058` warning if any custom rules depend
on `compile_commands.json` (e.g. to trigger rerunning clang-tooling
utilities).
Fixes: #16444
The internal tool "cmake_ninja_depends" now supports reading the list of ddi
files from a reponse file to circumvent Windows command line length limits.
Use this response file for dyndep rule on Windows.
Previously, given two libraries, X and Y where X depends on Y, all
object compilations of X would require the Y library to have been linked
before being compiled. This is not necessary and can instead be loosened
such that object compilations of X only depend on the order-only
dependencies of Y to be completed. This is to ensure that generated
sources, headers, custom commands, etc. are completed before X starts to
compile its objects.
This should help build performance in projects with many libraries which
cause a deep library dependency chain. Previously, a library at the
bottom would not start compilation until after all other libraries
completed, but now only its link step needs to wait and its compilation
jobs can be run in parallel with other tasks.
Fixes: #15555
Move the logic to cmSystemTools to be shared among the generators.
Revise the implementation and add comments justifying each possible
source for a limit.
The targets added by commit v3.6.0-rc1~240^2~2 (Ninja: Add `$subdir/all`
targets, 2016-03-11) use as `$subdir` the relative path from the top of
the source tree to the current source directory. This is not correct
when using `add_subdirectory(test test_bin)`. Instead we need to use
the relative path from the top of the binary tree to the current binary
directory as was done for related targets by commit v3.7.0-rc1~268^2
(Ninja: Add `$subdir/{test,install,package}` targets, 2016-08-05).
Diagnose failure to run `ninja --version` and abort early. Otherwise we
end up aborting with a confusing message about ninja version "" being
too old.
Closes: #16378
Add a cmOutputConverter to the cmLinkLineComputer and factory methods to
facilitate shell escapes.
Add state to the cmLinkLineComputer to record whether outputting for
response files or for watcom, to satisfy the cmOutputConverter API.
These are constant for the lifetime of the cmLinkLineComputer, even when
its functionality is extended in the future. This also keeps the
signatures of cmLinkLineComputer relatively simple.
Pass the cmComputeLinkInformation as a method parameter so that
cmLinkLineComputer is free from target-specific state. An instance
should be usable for all targets in a directory.
CMake has several classes which have too many responsibilities.
cmLocalGenerator is one of them. Start to extract the link line
computation. Create generator-specific implementations of the interface
to account for generator-specific behavior.
Unfortunately MSVC60 has different behavior to everything else and CMake
still generates makefiles for it. Isolate it with MSVC60-specific
names.
Per-source copyright/license notice headers that spell out copyright holder
names and years are hard to maintain and often out-of-date or plain wrong.
Precise contributor information is already maintained automatically by the
version control tool. Ultimately it is the receiver of a file who is
responsible for determining its licensing status, and per-source notices are
merely a convenience. Therefore it is simpler and more accurate for
each source to have a generic notice of the license name and references to
more detailed information on copyright holders and full license terms.
Our `Copyright.txt` file now contains a list of Contributors whose names
appeared source-level copyright notices. It also references version control
history for more precise information. Therefore we no longer need to spell
out the list of Contributors in each source file notice.
Replace CMake per-source copyright/license notice headers with a short
description of the license and links to `Copyright.txt` and online information
available from "https://cmake.org/licensing". The online URL also handles
cases of modules being copied out of our source into other projects, so we
can drop our notices about replacing links with full license text.
Run the `Utilities/Scripts/filter-notices.bash` script to perform the majority
of the replacements mechanically. Manually fix up shebang lines and trailing
newlines in a few files. Manually update the notices in a few files that the
script does not handle.
Create an internal `cmake -E cmake_ninja_dyndep` tool to read the "ddi"
files generated by `cmake -E cmake_ninja_depends` from all sources in a
target and generate a ninja dyndep file that tells Ninja about Fortran
module dependencies within the target and on target dependencies.
Create an internal `cmake -E cmake_ninja_depends` tool to scan an
already-preprocessed Fortran translation unit for modules that it
provides or requires. Save the information in a "ddi" file with a
CMake-private format for intermediate dynamic dependency information.
This file may the be loaded by another tool to be added later.
Kitware maintains a branch of Ninja with support for dynamically
discovered dependencies (dyndep) that has not yet been accepted
upstream. Add an internal API to check whether the Ninja version in use
for the build supports this feature.
Delay rejection of Fortran until after we've determined the version of
the `ninja` tool to be used. This will later allow us to enable Fortran
support based on the version of ninja.
While at it, make the rejection an immediate fatal error. Also provide
a stack trace so readers know what code tried to enable Fortran.
Ninja 1.7 introduced support for implicit outputs on build statements.
Add an internal API to check whether the Ninja version in use for the
build supports this feature.
Ninja 1.7 introduced support for implicit outputs on build statements.
Teach WriteBuild to generate the corresponding syntax. Leave it up to
callers to decide whether implicit outputs are supported by the Ninja
version in use. For now simply update all call sites to pass an empty
list of implicit outputs.
4332131d Convert: Make variables a bit more clear
5aca066c Convert: Remove UNCHANGED enum value
146bf926 Convert: Remove 'FULL' conversion
58ba87f8 Convert: Replace Convert(FULL) with equivalent
e80314d7 Ninja: Replace ternary with if()
563ac22a Convert: Replace trivial conversion with new method
08be47cf Convert: Replace UNCHANGED conversions with new API call
564d3a1d Convert: Extract ConvertToRelativePath from Convert()
95a659f1 Convert: Replace FULL conversions with equivalent
a8c7ccb1 VS: Replace FULL/UNCHANGED conversion with equivalent
5ad25ef4 Convert: Remove NONE conversion
ac463841 Convert: Replace uses of Convert(NONE)
998d9ee9 VS: Replace variable with an if()
ee49f006 Makefiles: Replace ternaries with if()s
51f7dcb0 Makefiles: Inline MakeLauncher into only caller
ba4ba7c3 Makefiles: Simplify MakeLauncher return value
...
