title: Python Web Scraping with search and non dynamic URI
body: <p>I'm a begginer in the world of python and web-scrapers, i am used to make scrapers with dynamic URLs, where the URI change when i input specific parameters in the URL itself.<br />
  Ex:  Wikipedia.<br />
  (if i input a search named &quot;Stack Overflow&quot; i will have a URI that looks like this: <a href="https://en.wikipedia.org/wiki/Stack_Overflow" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Stack_Overflow</a>)</p>
  <p>At the moment i was challenged to develop a web-scraper to collect data from <a href="http://comprasnet.gov.br/acesso.asp?url=/ConsultaLicitacoes/ConsLicitacao_texto.asp" rel="nofollow noreferrer">this page</a>.</p>
  <p>The field <strong>&quot;Texto/Termos a serem pesquisados&quot;</strong> corresponds a search field, but when i input the search the URL stays the same not letting me to get the right HTML code for my research.</p>
  <p>I am used to work with BeautifulSoup and Requests to do the scraping thing, but in this case it has no use, since the URL stays the same after the search.</p>
  <pre><code>import requests
  from bs4 import BeautifulSoup
  
  url = 'http://comprasnet.gov.br/acesso.asp?url=/ConsultaLicitacoes/ConsLicitacao_texto.asp'
  html = requests.get(url)
  bs0bj = BeautifulSoup(html.content,'html.parser')
  
  print(bsObj)
  # And from now on i cant go any further  
  
  </code></pre>
  <p>Usually i would do something like</p>
  <pre><code>url = 'https://en.wikipedia.org/wiki/'
  input = input('Input your search :)
  search = url + input
  </code></pre>
  <p>And then do all the BeautifulSoup thing, and findAll thing to get my data from the HTML code.</p>
  <p>I have tried to use Selenium too, but im looking for something different than that due to all the webdriver thing. With the following piece of code i have achieved to some odd results but i still cant scrape the HTML in a good way.</p>
  <pre><code>from selenium import webdriver
  from selenium.webdriver.common.keys import Keys
  import requests
  from bs4 import BeautifulSoup
  
  # Acess the page and input the search on the field
  
  driver = webdriver.Chrome()
  driver.get('http://comprasnet.gov.br/acesso.asp?url=/ConsultaLicitacoes/ConsLicitacao_texto.asp')
  driver.switch_to.frame('main2')
  busca = driver.find_element_by_id(&quot;txtTermo&quot;)
  busca.send_keys(&quot;GESTAO DE PESSOAS&quot;)
  #data_inicio = driver.find_element_by_id('dt_publ_ini')
  #data_inicio.send_keys(&quot;01/01/2018&quot;)
  #data_fim = driver.find_element_by_id('dt_publ_fim')
  #data_fim.send_keys('20/12/2018')
  botao = driver.find_element_by_id('ok')
  botao.click()
  
  </code></pre>
  <p>So given all that:</p>
  <ul>
  <li>There is a way to scrape data from these static urls ?</li>
  <li>Can i input a search in the field via code ?</li>
  <li>Why cant i scrape the right source code ?</li>
  </ul>