title: How do I scrape TLS certificates using go-colly?
body: <p>I am using <a href="https://github.com/gocolly/colly" rel="nofollow noreferrer">Colly</a> to scrape a website and I am trying to also get the TLS certificate that the site is presenting during the TLS handshake. I looked through the documentation and the response object but did not find what I was looking for.</p>
  <p>According to the docs, I can customize some http options by <a href="https://go-colly.org/docs/introduction/configuration/" rel="nofollow noreferrer">changing the default HTTP roundtripper</a>. I tried setting custom GetCertificate and GetClientCertificate functions, assuming that these functions would be used during the TLS handshake, but the print statements are never called.</p>
  <pre class="lang-golang prettyprint-override"><code>    // Instantiate default collector
      c := colly.NewCollector(
          // Visit only domains: hackerspaces.org, wiki.hackerspaces.org
          colly.AllowedDomains(&quot;pkg.go.dev&quot;),
      )
  
      c.WithTransport(&amp;http.Transport{
          TLSClientConfig: &amp;tls.Config{
              GetCertificate: func(ch *tls.ClientHelloInfo) (*tls.Certificate, error) {
                  fmt.Println(&quot;~~~GETCERT CALLED~~&quot;)
                  return nil, nil
              },
              GetClientCertificate: func(cri *tls.CertificateRequestInfo) (*tls.Certificate, error) {
                  fmt.Println(&quot;~~~GETCLIENTCERT CALLED~~&quot;)
                  return nil, nil
              },
          },
      })
  
  </code></pre>
  <p>Please help me scrape TLS certificates using Colly.</p>