The year before last, I tried to convince the framework group to distribute the framework library as a Nuget package. One of the most fatal problems was that the Nuget package did not have source debug support and was superior to other solutions in terms of distribution and package management. The final compromise choice is to use the source code direct reference project way, this scheme is not very beneficial to the development of the new branch of the framework library, in the source code protection is completely no guarantee, but in the context of the time, it is also one of the acceptable scheme. After years of development, Microsoft has made great strides in these areas, so follow me and see if you can solve the doubts in your mind?

1. PDB with a long history

When we see someone shiny and radiant, we all feel the urge to explore their past. Yeah, why should he get it and we can’t? Wouldn’t it be nice if we got up and explored his little secret, and maybe we could find something we don’t know about him, and provide some reference for our rise?That’s a bit of a digression. Let’s get back to business.

1.1 PDB and symbol files

The full name of PDB is Program Database. It is a debugging symbol file storage format developed by Microsoft. In Windows system, to debug DLL or EXE files, a symbol file (Symbols file) is required to support debugging. Symbolic files hold multiple pieces of data that are not really needed when running binaries, but can be useful during debugging. In general, a symbol file may contain:

  • The global variable
  • A local variable
  • The function name and the address of its entry point
  • Frame pointer elision (FPO) records
  • The source line number

When debugging, you must ensure that the debugger has access to the symbol file associated with the target you are debugging. Symbols are required for both real-time debugging and debugging crash dumps. You must get the correct symbols for the code to be debugged and load them into the debugger. The PDB file is the Windows reserved symbol file format. For those of you who are familiar with C++, this file should be familiar. It comes with Visual Studio and WinDbg and is one of the oldest file formats. What, want to see what the PDB file holds? Well, just to satisfy your curiosity, there is a DBH widget in the Windows debugger that can view the contents of PDB files.

mysymbols [1000000]: symopt 2 -

Symbol Options: 0x14c13
Symbol Options: 0x14c11

mysymbols [1000000]: addr 102cb4e

_MyFunction1@4
  name : _InterlockedIncrement@4
  addr :  102cb4e
  size : 0
 flags : 0
  type : 0
modbase :  1000000
 value :        0
   reg : 0
 scope : SymTagNull (0)
   tag : SymTagPublicSymbol (a)
 index : 2ab  
Copy the code

Of course, symbol files themselves come in many formats, such as COFF, which is the most commonly used debug symbol file format on Unix.

1.2 PDB variants – portable PDB

In order to adapt to the requirements of cross-platform, Microsoft has proposed the Portable PDB(PDB) standard to distinguish it from the long-used Windows PDB. It mainly supports managed code in.NET. Historically, the Windows PDB was used to store debugging information for both native and managed code, and its tools for reading and writing these PDB’s were supported only on the Windows platform. Portable PDB is designed to store managed debug information efficiently in a platform-independent format, has a wealth of support tools across multiple platforms, stores managed debug information in a portable format and generates a smaller PDB, which is also an important advantage when considering distribution size.

2 Use symbols to debug

2.1 Want to debug tripartite libraries or.net frameworks

Let’s look at an example. Sometimes you want to go into the framework to see what’s going on, especially if something unexpected happens. Suppose you set a breakpoint as shown below. So when you press F11 to get inside the framework, the code executes directly below, and this is what you see.By default, Visual Studio only debugs applicationsStep through code. This is a very useful feature because you usually want to understand and exploreThe code I wroteThe logic. Paying attention to yourself is the consciousness that gradually grows in the stage of learning language in life. Therefore, it starts from humanity, which is the best Feature! The functionality that enables this experience is aptly calledJust my code(” Just my Code “).

At some point, you want to debug the logic of a third-party component or the platform itself, and until then, debugging is very difficult. There are two main difficulties:

  1. Missing symbols for third-party components or platform binaries;
  2. Missing third-party component or platform binaries matching symbols associated with the source file.

JavaScript, by contrast, has the same properties as. NET has almost the opposite problem. The JavaScript community (both browsers and Node.js variants) uses SourceMap, which provides a great experience for debugging third-party lean code. However, JavaScript editors cannot provide an “just my code” experience.

For.net Core developers, we want to be able to easily and naturally switch between the default “just my code” experience and debugging with third-party components and platform sources. This is not a dream!

2.2 Debug. Net Core platform code

So how can we debug tripartite libraries or.net frameworks?

Visual Studio 2017 already supports symbolic debugging. Under the VS Tools menu, select Options/Debug/General TAB and configure the following parameters:

  • Disable “Enable My Code Only”
  • Enabling source links

Select the Options/Debug/Symbols TAB and set the following parameters:

  • Select Microsoft symbolic Server;
  • Select the NuGet.org symbol server (if you are debugging the Nuget library);
  • Set a directory in the cache symbol directory to avoid multiple downloads of the symbol library.

If you are using VS Code, you can configure the debugger setting: launch.json for each project

"justMyCode": false."symbolOptions": {
    "searchMicrosoftSymbolServer": true."searchNuGetOrgSymbolServer": true
},
"suppressJITOptimizations": true."env": {
    "COMPlus_ZapDisable": "1"."COMPlus_ReadyToRun": "0"
}
Copy the code

Note:

  1. Not every library on nuget.org indexes its.pDB files. If you find that the debugger cannot find the PDB file for the open source library you are using, encourage authors to upload their PDB.
  2. Only microsoft-supplied libraries will have their.pDB files on Microsoft symbolic servers, so you can disable this option if you are only interested in tripartite libraries.

Start debugging, find VS start to download the symbol file, after downloading, enter the breakpoint. When we press F11, the following screen pops up:

3. SourceLink

SourceLink is a productivity feature for developers that allows a set of software packages and specifications to be embedded in the PDB during compilation with unique information about an assembly’s original Source code, metadata added to PDB files via SourceLink, A mapping relationship is established with local source code files and code files in the repository.

So Visual Studio debugging can download files as needed and provide users with source code debugging, Microsoft libraries (e.g. Source Link has been enabled for NET Core and Roslyn.

3.1 Why SourceLink?

Most debugging is done against source code built locally on the developer’s computer. In this case, matching the binaries with the source code is not difficult.

However, in many debugging scenarios, the original source code is not immediately available. Two good examples of this are debug crash dumps or third-party libraries. In these cases, it can be very difficult for a developer to obtain the exact source code (possibly the specific version) that was built to generate the binaries being debugged. Source Link solves this problem by embedding unique information about the Source code in the PDB, such as git commit hashes. Diagnostic tools, such as debuggers, can use this unique information to retrieve the original source code from a managed service, such as GitHub.

The original version of Sourcelink was implemented by @Ctaggart and is now archived and has been added to the.NET team. Microsoft people worked with Ctaggart on the current version.

Official website: github.com/dotnet/sour…

3.2 File specification for SourceLink

SourceLink is a Json configuration file with the following content format:

{
    "$schema": "http://json-schema.org/draft-04/schema#"."title": "SourceLink"."description": "A mapping of source file paths to URLs"."type": "object"."properties": {
        "documents": {
            "type": "object"."minProperties": 1."additionalProperties": {
                "type": "string"
            },
            "description": "Each document is defined by a file path and a URL. Original source file paths are compared case-insensitively to documents and the resulting URL is used to download source. The document may contain an asterisk to represent a wildcard  in order to match anything in the asterisk's location. The rules for the asterisk are as follows: 1. The only acceptable wildcard is one and only one '*', which if present will be replaced by a relative path. 2. If the file path does not contain a *, the URL cannot contain a * and if the file path contains a * the URL must contain a *. 3. If the file path contains a *,  it must be the final character. 4. If the URL contains a *, it may be anywhere in the URL."}},"required": ["documents"]}Copy the code

To reduce the effort of generating this JSON, Microsoft provides a series of software packages that automatically generate Source Link files.

3.3 Automatically Generating SourceLink

In the.net Core project, add the following configuration to the. Csproj file

<Project Sdk="Microsoft.NET.Sdk">
 <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
 
    <! -- Optional: Publish the repository URL in the built .nupkg (in the NuSpec <Repository> element) -->
    <PublishRepositoryUrl>true</PublishRepositoryUrl>
 
    <! -- Optional: Embed source files that are not tracked by the source control manager in the PDB -->
    <EmbedUntrackedSources>true</EmbedUntrackedSources>
  
    <! -- Optional: Build symbol package (.snupkg) to distribute the PDB containing Source Link -->
    <IncludeSymbols>true</IncludeSymbols>
    <SymbolPackageFormat>snupkg</SymbolPackageFormat>
  </PropertyGroup>
  <ItemGroup>
    <! -- Add PackageReference specific for your source control provider (see below) --> 
  </ItemGroup>
</Project>
Copy the code

Refer to the following packages as needed. Note that this is set to PrivateAssets to avoid the project referencing the package downloading the Sourcelink package after publishing it as a Nuget package.

  • github.com and GitHub Enterprise
<ItemGroup>
  <PackageReference Include="Microsoft.SourceLink.GitHub" Version="1.0.0" PrivateAssets="All"/>
</ItemGroup>
Copy the code
  • Azure Repos (former Visual Studio Team Services)
<ItemGroup>
  <PackageReference Include="Microsoft.SourceLink.AzureRepos.Git" Version="1.0.0" PrivateAssets="All"/>
</ItemGroup>
Copy the code
  • Azure DevOps Server (former Team Foundation Server)
<ItemGroup>
  <PackageReference Include="Microsoft.SourceLink.AzureDevOpsServer.Git" Version="1.0.0" PrivateAssets="All"/>
</ItemGroup>
Copy the code
  • If your server configuration has the IIS virtual directory is not empty, please specify the directory in the SourceLinkAzureDevOpsServerGitHost project, as shown below
<ItemGroup>
  <SourceLinkAzureDevOpsServerGitHost Include="server-name" VirtualDirectory="tfs"/>
</ItemGroup>
Copy the code

The Include attribute specifies the domain and the port of the server (such as server-name or server-name:8080).

  • GitLab
<ItemGroup>
  <PackageReference Include="Microsoft.SourceLink.GitLab" Version="1.0.0" PrivateAssets="All"/>
</ItemGroup>
Copy the code
  • Bitbucket
<ItemGroup>
  <PackageReference Include="Microsoft.SourceLink.Bitbucket.Git" Version="1.0.0" PrivateAssets="All"/>
</ItemGroup>
Copy the code

If your project is from version 4.7 before Bitbucket Server or Bitbucket Data Center hosting SourceLinkBitbucketGitHost, besides package reference, also have to specify the project team:

<ItemGroup>
  <SourceLinkBitbucketGitHost Include="bitbucket.yourdomain.com" Version="4.5"/>
</ItemGroup>
Copy the code

Project team SourceLinkBitbucketGitHost Bitbucket host specified domain and Bitbucket version. This release is important because the URL format used to access files changes with version 4.7. By default, the source link is in the new format (version 4.7+).

  • gitweb (pre-release)
<ItemGroup>
  <PackageReference Include="Microsoft.SourceLink.GitWeb" Version="1.1.0 - beta - 20204-02" PrivateAssets="All"/>
</ItemGroup>
Copy the code

The developer must choose to generate the source link file. Urls contained in these files may point to private source repositories that are not intended to be exposed to anyone with access to symbol files, so developers should choose wisely.

  • Open source developers often choose not to participate in source link file generation because they usually don’t have publicity issues.
  • Corporate developers should also choose to select source link file generation using the following methods:

All application assets (binaries, symbols, and source code) are used within the corporate firewall, so only users who have access to these assets can see them. Binary assets are shipped from outside, but symbols and sources are used only within the corporate firewall, so only users with access to symbols and source assets can view them.

  • Corporate developers have another option, arguably an anti-pattern, as follows:

Binary and symbolic assets are shared externally. Symbolic assets contain source link files (and possibly generated file assets). Source link files point to symbolic sources that require authentication, such as VSTS. Authorized users, which may be small in number, will have access to the source. Unauthorized users (possibly more numerous) receive access denial messages from endpoints they do not understand.

3.4 source embedded

In some cases, it is beneficial to embed source code in symbols so that you can easily deploy the source code for debugging. However, this is a trade-off between convenience and PDB size. Although the source compression is stored in a PDB that contains many source files, it can greatly increase the size of the PDB.

The following embedding options (for Windows and portable PDB) are available:

  • Embed only source files that are not tracked by source control (for example, files generated during build). The rest of the files are mapped from the source link.
  • Embed a manually selected subset of source files.
  • Embed all source files

EmbedAllSources The Project attribute with a Boolean value means that all sources passed to the compiler should be embedded in the PDB.

Automatic embedding of untraced source file source links enables debuggers and other tools to find the source content of files tracked by source controls. However, not all files involved in the build are tracked. For example, files generated during build are usually not checked into the repository. Although you can manually identify such files and tag them to embed them in the PDB, the process is cumbersome and error-prone.

The Sourcelink.Embed project already supports automatic identification of files embedded instead of tracking them by source control. These apis identify files that are not tracked by source control (for example, for a Git repository, match the itemized files in a.gitignore file) and set up EmbedUntrackedSources, which then instruct the compiler to embed the untracked source code. Here is an example of embedding source code, adding the following configuration to the.csproj project file

<PropertyGroup> 
  <EmbedAllSources>True</EmbedAllSources>
  <! -- <EmbedUntrackedSources>true</EmbedUntrackedSources> -->
</PropertyGroup>
Copy the code

4. Publish the symbol file

By default, symbols are built as separate files to minimize the size of the binaries. These files need to be published by the system that built them, such as a CI server, and discovered and retrieved by the system that needs them, such as a debugger.

4.1 Publish symbols to the symbol server

Today, symbol servers are primarily used to host symbol files in enterprise environments. Symbol server tools are available for such environments and have recently been integrated into VSTS and exposed as a service.

For developers publishing their libraries on NuGet.org, the options for publicly available symbol servers are limited, and the process of publishing and using symbols is much more complicated than it should be. As a result, the number of easy-to-debug packages released on NuGet.org is small.

Symbol packages (SNUPkg) Today, symbol packages are used to distribute symbols and sources. A good debugging experience depends on the presence of debug symbols because they provide critical information such as the association between compiled code and source code, the names of local variables, stack traces, and so on. You can use the symbol package (.snupkg) to distribute these symbols and improve the debugging experience of NuGet packages.

If dotnet CLI or MSBuild is used, in addition to.nupkg files, set IncludeSymbols and SymbolPackageFormat properties to create.snupkg files.

There are several ways to create.snupkg files to implement this requirement.

  • Add the following properties to the.csproj file:
<PropertyGroup>
    <IncludeSymbols>true</IncludeSymbols>
    <SymbolPackageFormat>snupkg</SymbolPackageFormat>
</PropertyGroup>
Copy the code
  • Specify these properties on the command line:
dotnet pack MyPackage.csproj -p:IncludeSymbols=true -p:SymbolPackageFormat=snupkg
Copy the code
  • Or use Msbuild
msbuild MyPackage.csproj /t:pack /p:IncludeSymbols=true /p:SymbolPackageFormat=snupkg
Copy the code
  • Use nuget
nuget pack MyPackage.nuspec -Symbols -SymbolPackageFormat snupkg
nuget pack MyPackage.csproj -Symbols -SymbolPackageFormat snupkg
Copy the code

Important: My thoughts and experience

Here is an important experience, if you want to get it, you need to visit my CSDN blog, sorry, can not directly paste, because here is paid content exclusive. If I had offered it earlier, I would have won, right?

5. Use PDB (symbols) and SourceLink

It should be convenient to use symbols in Visual Studio or other tools. It needs to apply to all. NET implementation, including. NET Core,.NET Framework, Xamarin, Unity and UWP.

There are three main scenarios for debugging using symbols:

  • Debug the application during development. In this case, third-party library symbols need to be retrieved outside of the build system, while application symbols need to be born as part of the build.
  • Debug the application in the deployment state. In this case, you need to retrieve the application and library symbols. This scenario will attach to the running application.
  • Debug the fault dump of the application. In this case, application and library symbols are required.

5.1 Symbols and binaries are distributed together

This solution is suitable for the development phase. Does not apply if the symbol is already embedded in the binary. NuGet is a common case for binary distribution, allowing symbols to be deployed with binaries.

The key feature is that the symbol file will exist directly next to the loaded code file on disk, making it easy for the debugger to find a matching symbol file for a given binary. According to the instructions in this document, binaries and symbols are usually juxtaposed in the NuGet package in a structure similar to the following example.

/ /lib /netstandard2.0 foo.dll foo.pdb.NET Core development is NuGet centric, which helps solve this problem. During development (for example, using dotnet Run), the.NET Core runtime loads libraries from the NuGet cache by default, allowing the debugger to look for matching symbol files in the same place.

The.NET Framework uses NuGet for development, but less formally. When a project is built, the NuGet library is copied to the application bin directory instead of symbols. As a result, the link between the code binary and the symbol is lost. Build systems should use the following logic to better enable signed debugging:

If you copy the code binary to a location, you also copy the symbol file to the same location. In most cases, this will be the application bin directory.

5.2 Symbols and binary files are distributed separately

This solution is suitable for debugging deployed applications, crash dumps, and debugging third-party libraries that are not attached to symbols in NuGet packages. In these cases, you need to get the symbol from the symbol server.

As mentioned above, we recommend a common symbol service for symbols uploaded to NuGet.org. You can also use other symbol servers as needed.

5.3 Obtaining and Consuming Source Files

Getting and consuming source files is primarily about getting symbols in the first place. With symbols, the debugger will find that one or more of the following cases are true:

  • The symbol file contains the embedded source of the source file the debugger is looking for, which the debugger will use at this point.
  • The symbol file is Windows PDB and contains embedded source server information. You can use a build tool such as PDBSTR or GitLink to modify an existing Windows PDB with source server information.
  • The symbol file contains an embedded source link file, where the debugger interprets and executes declarations in the source link file.
  • Otherwise, sources will not be available through the mechanisms discussed in this document

Note: If the debugger cannot find the source file in the symbol file or through the source link, it can still try to find it on the symbol server using the ** simple symbol query protocol **. This will allow for situations where the source is available later after the binaries and symbol builds are complete.

6 summary

Understand SourceLink relationship with PDB, it cost me a New Year’s day holiday, had planned on vacation within the article brief introduction, found that, in the case of themselves are not clear, write these is to everybody and their irresponsible, as a result, or calm down and carefully analyze each link, the hope can help to you.