[dump_breakpad_symbols] Have dump_breakpad_symbols produce an archive(2)
Added some logic to enable soft trasition
[dump_breakpad_symbols] Have dump_breakpad_symbols produce an archive
The go tool itself should do the tarring, as it easiest to do there where
the tool has the relevant information; this would save the recipe from
having to make brittle assumptions. If the build produced the archive directly,
the recipe would have to do no more work beyond just uploading it, to the same downstream effect.
Test: ninja -C out/x64 build/gn:breakpad_symbols #The desired tarball is
[mdns] Adds BUILD.gn for mDNS code.
TEST: built and run with client impl (see below)
The code using this can be found here:
[mdns] Implement basic mDNS lookup
We need to replace netaddr with something else. This library
attempts to implement mDNS in a OS agnostic way for use as
a host tool or in botanist.
[botanist] Adding support for host-cmds
This enables ACTS tests to be run in swarming.
Bug: IN-775 #comment
[pdu][amt] Use the correct power state
Don't read power state from the command line, this is a leftover
[botanist] Move LogListener start to before paving
This will allow us to get logs from the paving process.
[botanist] Tar in subroutine
Tar results in a subroutine while busy-printing so that we do not hit
Swarming's i/o timeout when handling large files (as we are currently
doing with the Vulkan Conformance Suite).
[serial] Provide empty implementation for Darwin
This unblocks building botanist for macOS.
Move botanist to tools
This change moves all of botanists implementation from infra/infra
to tools project repository.
[digest] Support for HTTP Digest Authentication
This package implements support for HTTP Digest Authentication as a
RoundTripper interface implementation.
This is based on the code.google.com/p/mlab-ns2/gae/ns/digest package.
[symbolize] Check whether the path is absolute
This avoids duplicating the path if it's already absolute.
[symbolize] Add ids.txt relative filepaths
Zircon uses paths reletive to the zircon working directory. Fuchsia uses
absolute paths. We also need to support relocatable out directories. To
do this I've added a flag -ids-rel. That does this.
Bug: TC-290 #comment Makes symbolizer understand ids.txt rel paths
[symbolize] Fix bug in backtrace presenter
There was a bug in the backtrace presenter where header was assumed to be
non-nil. This caused a null reference exception when I tried to test kernel
Bug: ZX-2060 #comment Fix symbolize to handle no headers
[symbolize] Make syslog and loglistener nicer
These changes just ensure that when using syslog
and loglistner, the output from symbolizer doesn't
yell too much at you and looks nice.
1. Backtraces will sometimes contain a pc 0x0 case. It is not trivial
to stop this from happening. To avoid yelling at the user too much
we handle this case specifically.
2. It is sometimes the case that names for modules like <VMO#1234=ld.so>
are used for module names in the markup. When this happens twice
it will cause a name conflict. This change removes the warning
for this because it is too loud. A TODO is added to account for it.
3. When using syslog headers there is also a type at the end like "INFO:"
or "WARN:". The symbolizer was treating this as part of the message.
This was fine at the time it was added but now for certain aesthetic
things to happen some tags need to be on their own line with no
other content. To make this happen we need to parse the type. This
ensures that backtraces are printed properly and that context is
[symbolize] Fix bug regarding leading zeros.
This patch removes leading zeros from numbers before parsing them.
This fixes an unknown bug where process ids and thread ids were
probably being handle correctly in the sense that they were unique
but were not being handled correctly in the sense that they were
being mapped to incorrect values. This also eleminates a possible
panic that was found in TC-273 by Todd where due to lossy uart an
'x' was dropped resulting in an '0x8' turning into '08' which
caused the panic because strconv.ParseUint interpeted that number
as octal due to the leading zero even though '8' is not an octal
digit (hence the panic).
[bloaty] Implement a proper semaphore around the goroutines
[bloaty] Improve logging
[bloaty] Don't close the pipe if it didn't get created
I was overaggressive on the closing :/
[bloaty] Close pipe only on failures
Only close the pipe when the command fails, otherwise the call to Wait()
should take care of it.
[bloaty] Explicitly close StdoutPipe()
[bloaty] Pass file to goroutine closure
Might help address the -too-many-open-files issue
[bloatalyzer] Fixing hangs
Replaces the ReadCSV goroutine and associated channel with a function
that simply passes along data, as I suspect the intermittent failures
we saw were an issue with the StdoutPipe never hitting the EOF.
Also wraps the data processing in a goroutine so we should not deadlock
[tools] Switch to go modules
[covargs] Use tmp files for llvm-cov/llvm-profdata output
[covargs] Filter uninstrumented ELFs
[covargs] Fix minor things so coverage works
[covargs] Don't fail if a profile is missing
[symbolize] Handle error that I forgot to handle
[covargs] Don't error out multiple profiles per test
[symbolize] Use Create rather than Open for json-output
[symbolize] Reverse order of inline backtraces
I got the order for how inlines are given backwards. This
change updates the symbolizer to display inlined calls
in the correct order.
[elflib] Trim whitespace from the end of filenames in ids.txt
We hit an issue where a space at the end of a filename caused a
fatal error. This is not really the sort of thing that should
be causing symbolizer to fail at all but this is good duct tape
to solve the problem. It's probably good to do this anyway as it
increases the number of cases in which we can successfully symbolize.
[covargs] Introduce covargs tool
This tool takes in many sources of information, merges them,
then allows many commands to be executed based on that information.
It's like xargs but for coverage information. See ExampleMain for
details on how to run it.
[bloaty] Limit number of concurrent bloaty runs
[bloaty] Color html output by count
[bloatalyzer] Add HTML output
This takes ~34 seconds on my workstation.
[symbolize] Add newline at end of lone backtrace.
This is a dumb and shameful bug that took all day to find
because rollers are slow. I didn't put a newline at the end
so these things are not printing out correctly.
[bloatalyzer] Refactoring data structure
So that it's more useful for manipulating.
[symbolize] Fix dropped backtrace bug
Backtraces were begin seeminglly dropped from output because
they were missing symbolization info. This was caused by the
code that output strings looping over source locations. If
symbolization failed however that array is empty. This
change adds a fallback for that case to output what information
is still possible.
[ninjatrace] Normalize json flag
[symbolize] Add better backtrace formatting
This change adds a new backtrace formatter that intercepts lone
backtrace lines and prints them out on multiple lines with all
information we've thusfar determined would be nice. inlines are
printed out on seperate lines in the same format with a special
[symbolize] Check modtime on ids.txt
This change checks the mod time on the ids.txt file
prior to reading the file and thus prior to performing
verificiation of the ids.txt. Coupled with caching this
reduces the primary cost of the symbolize to be parsing (30-40%, 30% regex alone),
lock contention (15%). This represents something like a 1000x
speed up. Further optimization seems much harder. Increasing
go channel buffer sizes should give a 25%-50% improvement.
[bloatalyzer] Renaming bloaty to bloatalyzer
So as not to be confused with the actual bloaty binary.
[symbolizer] Fix faulty return in cache code
I put a 'return' where I should have put a 'continue'.
This causes the symbolizer to just stop symbolizing after a cache hit.
That is not very functional.
[symbolize] Cache llvm-symbolizer results
This change handles the spew case that causes critical slow
downs when processing the symbolizer spew from bad asan output.
Right now on 16MB inputs mostly filled with backtraces the filter
is grinding to a halt as a result of all the backtracing. This
change should massively speed up that case.
[cache] Make sure LRUCache actually implements the interface
Update the test to enforce that.
[cache] Simple LRU cache implementation
This is intended for use with the symbolizer.
[symbolize] Decompose presentor using new pipeline abstraction
This change introduces a new pipeline abstraction that allows
many parts of the pipeline to be seperated allowing for a minimum
of concerns on the pipeline code itself. This change uses it to
improve the presentor but I plan to use this for Filter, Demuxer,
a parser, etc... For instance Filter will be split into a line
parser and an annotator. Demuxer will remain more or less the
same. Many features such logging and keeping track of certian
error states can be added just as parts of the pipeline.
For instance 'modNamesByBuildID' exists just to keep track of
a certain kind of error. This can just be its own node in the
pipeline and can then collect data more neatly. This will also
make adding line-splitting/merging quite easy in the future.
[symbolize] Support triggers