title: <input values missing when parsing html with Jsoup?
body: <p>I am trying to scrap webpages with Jsoup. Jsoup doesn't seem to capture the <code>&lt;input</code> elements like Chrome does.</p>
  
  <p>It is missing values such as these:</p>
  
  <pre><code>&lt;input type=​"hidden" id=​"fileId" value=​"3168935269"&gt;
  &lt;input type=​"hidden" id=​"secondsLeft" value=​"20"&gt;​​
  </code></pre>
  
  <p>Using Jsoup I extracted these elements:</p>
  
  <pre><code>&lt;input type="hidden" class="jsItemDirId" value="yRg1N-QP" /&gt;
  
  &lt;input type="hidden" class="jsItemFileId" value="i-EbooI0" /&gt;
  
  &lt;input type="hidden" id="fbAppId" value="255519317820035" /&gt;
  
  &lt;input type="hidden" id="sPrefix" value="http://search.4shared.com" /&gt;
  
  &lt;input type="hidden" class="sLink file" value="/q/CCAD/1" /&gt;
  
  &lt;input type="hidden" class="sLink video" value="/q/CCQD/1/video" /&gt;
  
  &lt;input type="hidden" class="sLink music" value="/q/CCQD/1/music" /&gt;
  
  &lt;input type="hidden" class="sLink photo" value="/q/CCQD/1/photo" /&gt;
  
  &lt;input type="hidden" class="sLink games" value="/q/CCQD/1/game" /&gt;
  
  &lt;input type="hidden" class="sLink book" value="/q/CCQD/1/books_office" /&gt;
  
  &lt;input type="hidden" class="sLink featured_videos" value="/q/CCQD/1/video" /&gt;
  
  &lt;input type="hidden" id="sBreadcrumbsPhrase" value="Searching" /&gt;
  
  &lt;input type="text" id="searchQuery" placeholder="Search files" /&gt;
  
  &lt;input type="hidden" id="interval" value="600000" /&gt;
  
  &lt;input type="hidden" id="archiveReadyDownload" value="Your file is ready for download:" /&gt;
  
  &lt;input type="hidden" id="defAvatar" value="http://static.4shared.com/images/user2.png?ver=2906097813" /&gt;
  
  &lt;input type="hidden" id="zipAvatar" value="http://static.4shared.com/icons/32x32/zip.png?ver=655479399" /&gt;
  
  &lt;input type="hidden" id="b1Avatar" value="http://static.4shared.com/icons/32x32/b1.png?ver=703417425" /&gt;
  
  &lt;input type="hidden" id="torrentAvatar" value="http://static.4shared.com/icons/32x32/torrent.png?ver=1628575404" /&gt;
  
  &lt;input type="hidden" id="contactRequestText" value="Your friend $[p1] just joined 4shared." /&gt;
  
  &lt;input type="button" value="Ok" onclick="checkAndStartDownload(event);" style="width:80px" /&gt;
  
  &lt;input type="button" value="Cancel" onclick="hideTermsOfUse();" /&gt;
  
  &lt;input type="hidden" id="startTitle" value="Share" /&gt;
  
  &lt;input type="hidden" id="sharingFolderTitle" value="Share folder" /&gt;
  
  &lt;input type="hidden" id="sharingFileTitle" value="Share file" /&gt;
  
  &lt;input type="hidden" id="placeHolderEnterEmailAdresses" value="Enter names or e-mail addresses" /&gt;
  
  &lt;input type="hidden" id="dLinkPay" value="Direct link is available only for Premium Users.&amp;lt;br&amp;gt; Sign Up to premium account to get all 4shared Premium Features." /&gt;
  
  &lt;input type="hidden" id="premiumRequired" value="Premium account required!" /&gt;
  
  &lt;input type="hidden" id="hosted" value="Hosted at" /&gt;
  
  &lt;input type="hidden" id="fbInviteFolderTitle" value="I've shared a folder with you on 4shared. Find out what it is!" /&gt;
  
  &lt;input type="hidden" id="fbInviteFileTitle" value="I've shared a file with you on 4shared. Find out what it is!" /&gt;
  
  &lt;input type="hidden" id="contacts" value="Contacts" /&gt;
  
  &lt;input type="hidden" id="fb_share_folder_img" value="http://static.4shared.com/images/facebook/share_folder.png?ver=2422162001" /&gt;
  
  &lt;input type="hidden" id="fb_share_file_img" value="http://static.4shared.com/images/facebook/share_file.png?ver=1565381062" /&gt;
  
  &lt;input type="hidden" id="fb_redir_param" value="https://www.4shared.com/servlet/signin/facebook?fp=https://www.4shared.com/account/home.jsp" /&gt;
  
  &lt;input type="hidden" id="fileSuccessfullSent" value="Your file was successfully sent" /&gt;
  
  &lt;input type="hidden" id="folderSuccessfullSent" value="Your folder was successfully sent" /&gt;
  
  &lt;input type="hidden" id="fbRequestSharedText" value="I'd like to share $[p0] with you" /&gt;
  
  &lt;input type="hidden" id="fbSharingOff" value="null" /&gt;
  
  &lt;input type="hidden" id="fbInviteText" value="4shared.com - free web-based file sharing and storage." /&gt;
  
  &lt;input type="radio" class="readFlag" name="permissions" value="read" checked="checked" /&gt;
  
  &lt;input type="radio" class="writeFlag" name="permissions" value="write" /&gt;
  
  &lt;input class="lucida dark-gray selectable" id="simpleViewLink" type="text" readonly="readonly" /&gt;
  
  &lt;input type="text" id="emails" class="lucida f12 dark-gray tags gaClick" data-element="shF-2-1" name="emails" tabindex="3" /&gt;
  
  &lt;input type="radio" class="readFlag" name="permissions" value="read" checked="checked" /&gt;
  
  &lt;input type="radio" class="writeFlag" name="permissions" value="write" /&gt;
  
  &lt;input type="text" id="downloadFileLink" class="lucida f12 selectable" name="" tabindex="3" /&gt;
  
  &lt;input type="text" class="lucida f12 dark-gray selectable" name="" tabindex="4" value="" id="premiumDirectLink" /&gt;
  
  &lt;input type="text" class="lucida f12 selectable" id="fileHTMLembed" name="" tabindex="3" /&gt;
  
  &lt;input type="text" id="fileForumEmbed" class="lucida f12 selectable" name="" tabindex="4" /&gt;
  
  &lt;input type="text" class="lucida f12 selectable" id="fileEmbed" tabindex="5" /&gt;
  
  &lt;input class="lucida f12 dark-gray selectable" id="searchFriendsInput" type="text" placeholder="Search by name or e-mail address" /&gt;
  
  &lt;input id="tags_2" type="text" class="tags" /&gt;
  
  &lt;input type="radio" class="readFlag" name="permissions" value="read" checked="checked" /&gt;
  
  &lt;input type="radio" class="writeFlag" name="permissions" value="write" /&gt;
  
  &lt;input type="radio" class="readFlag" name="permissions" value="read" checked="checked" /&gt;
  
  &lt;input type="radio" class="writeFlag" name="permissions" value="write" /&gt;
  
  &lt;input type="text" class="lucida f12 ffshadow dark-gray" name="" tabindex="4" value="" id="subdomainInput" /&gt;
  
  &lt;input type="text" class="lucida f12 ffshadow dark-gray" name="" tabindex="3" value="" id="subdomainValue" readonly="true" /&gt;
  
  &lt;input type="hidden" id="allreadyPasswordProtectedMess" value="You can't set password for this folder, because the parent folder '$[1]' is password protected." /&gt;
  
  &lt;input type="hidden" id="passwordChangeConfirmTitle" value="Password Change" /&gt;
  
  &lt;input type="hidden" id="passwordChangeConfirmBody" value="Some child directory already password protected. &amp;lt;br/&amp;gt; Changing password of current directory will cause password overwrite on children's " /&gt;
  
  &lt;input type="hidden" id="confirmButtonMsg" value="Change" /&gt;
  
  &lt;input type="hidden" id="cancelButtonMsg" value="Cancel" /&gt;
  
  &lt;input type="text" class="passInput lucida f12" name="" tabindex="4" value="" id="passwordInput" /&gt;
  
  &lt;input type="password" class="passInput lucida f12" name="" tabindex="4" value="" id="changePasswordInput" readonly="true" /&gt;
  
  &lt;input type="hidden" id="previewLinkForEmbed" /&gt;
  
  &lt;input type="hidden" id="previewLinkForWidget" /&gt;
  
  &lt;input class="lucida f12 dark-gray" id="widget_width" type="text" style="width:30px;" /&gt;
  
  &lt;input class="lucida f12 dark-gray" id="widget_height" type="text" style="width:30px;" /&gt;
  
  &lt;input type="text" class="lucida f12 dark-gray selectable" name="" tabindex="3" id="htmlEmbed" /&gt;
  
  &lt;input type="text" class="lucida f12 dark-gray selectable" name="" tabindex="4" id="forumEmbed" /&gt;
  
  &lt;input type="text" value="http://www.4shared.com/android/i-EbooI0/batman_hd.html" readonly="readonly" onclick="this.focus();this.select()" class="field1 gaClick" data-element="16" dir="ltr" /&gt;
  
  &lt;input type="text" value="&amp;lt;a href=&amp;quot;http://www.4shared.com/android/i-EbooI0/batman_hd.html&amp;quot; target=_blank&amp;gt;batman hd.apk&amp;lt;/a&amp;gt;" readonly="readonly" onclick="this.focus();this.select()" class="field1 gaClick" data-element="17" dir="ltr" /&gt;
  
  &lt;input type="text" value="[URL=http://www.4shared.com/android/i-EbooI0/batman_hd.html]batman hd.apk[/URL]" readonly="readonly" onclick="this.focus();this.select()" class="field1 gaClick" data-element="18" dir="ltr" /&gt;
  
  &lt;input type="hidden" name="showComments" value="true" /&gt;
  
  &lt;input type="hidden" name="showPart" value="commentList" /&gt;
  
  &lt;input type="hidden" name="replyId" value="" /&gt;
  
  &lt;input type="hidden" id="norecaptcha" name="norecaptcha" value="" /&gt;
  
  &lt;input type="hidden" name="start" value="0" /&gt;
  
  &lt;input id="submitCommBtn" type="submit" value="Add New Comment" class="gaClick floatLeft f11 marginT10 round4 lucida no-line sendCommentButton" data-element="32" /&gt;
  
  &lt;input type="text" class="input-gray-big wide round4" id="recaptcha_response_field" name="recaptcha_response_field" style="width:250px" /&gt;
  
  &lt;input class="field2" id="submitCommBtn" type="submit" value="Confirm" /&gt;
  
  &lt;input type="text" name="fileName" value="4shared" class="xBox" /&gt;
  
  &lt;input type="hidden" name="newValue" value="" /&gt;
  
  &lt;input type="hidden" name="mode" value="" /&gt;
  
  &lt;input type="hidden" name="fid" value="3168935269" /&gt;
  
  &lt;input type="hidden" name="mode" value="3" /&gt;
  
  &lt;input type="hidden" name="fid" value="3168935269" /&gt;
  
  &lt;input type="submit" value="Save" class="bluePopupButton marginT15 round5 f12 floatLeft marginR10" /&gt;
  
  &lt;input type="button" value="Cancel" class="grayPopupButton marginT15 round5 f12 floatLeft" onclick="quickEditCancel(1)" /&gt;
  
  &lt;input type="hidden" name="mode" value="3" /&gt;
  
  &lt;input type="hidden" name="fid" value="3168935269" /&gt;
  
  &lt;input type="submit" value="Save" class="bluePopupButton marginT15 round5 f12 floatLeft marginR10" /&gt;
  
  &lt;input type="button" value="Cancel" class="grayPopupButton marginT15 round5 f12 floatLeft" onclick="quickEditCancel(1)" /&gt;
  
  &lt;input type="hidden" name="mode" value="3" /&gt;
  
  &lt;input type="hidden" name="fid" value="3168935269" /&gt;
  
  &lt;input type="submit" value="Save" class="bluePopupButton marginT15 round5 f12 floatLeft marginR10" /&gt;
  
  &lt;input type="button" value="Cancel" class="grayPopupButton marginT15 round5 f12 floatLeft" onclick="quickEditCancel(1)" /&gt;
  
  &lt;input type="text" name="newValue" class="xBox" style="width:200px" /&gt;
  
  &lt;input type="hidden" name="mode" value="2" /&gt;
  
  &lt;input type="hidden" name="fid" value="3168935269" /&gt;
  
  &lt;input type="submit" value="Save" class="bluePopupButton marginT15 round5 f12 floatLeft marginR10" /&gt;
  
  &lt;input type="button" value="Cancel" class="grayPopupButton marginT15 round5 f12" onclick="quickEditCancel(1)" /&gt;
  
  &lt;input type="text" name="newValue" class="xBox" style="width:330px" onkeypress="return quickEditIsValidCharForFileName(event);" /&gt;
  
  &lt;input type="hidden" name="mode" value="10" /&gt;
  
  &lt;input type="hidden" name="fid" value="3168935269" /&gt;
  
  &lt;input type="submit" value="Save" class="bluePopupButton marginT15 round5 f12 floatRight marginL10" /&gt;
  
  &lt;input type="button" value="Cancel" class="grayPopupButton marginT15 round5 f12 floatRight" onclick="quickEditCancel();" /&gt;
  
  &lt;input type="hidden" name="mode" value="3" /&gt;
  
  &lt;input type="hidden" name="did" value="0" /&gt;
  
  &lt;input type="submit" value="Save" class="bluePopupButton marginT15 round5 f12 floatLeft marginR10" /&gt;
  
  &lt;input type="button" value="Cancel" class="grayPopupButton marginT15 round5 f12 floatLeft" onclick="quickEditCancel(1)" /&gt;
  
  &lt;input type="text" name="searchName" style="width:250px;padding:1px 0" class="ajax-suggestion field gaClick" data-element="fs1" autocomplete="off" /&gt;
  
  &lt;input type="submit" name="submitButton" value="Search" class="button gaClick" data-element="fs3" /&gt;
  
  &lt;input type="hidden" name="searchmode" value="2" /&gt;
  </code></pre>
  
  <p>Using try.jsoup.com also did not yield these input types like Chrome which suggests that it is not my code but rather Jsoup.</p>
  
  <p>Reading through other threads suggest that Javascript may be changing the html after loading the webpage. There were no viable answers on how to fix this.</p>
  
  <p>What am I doing wrong and how do I fix it?</p>
  
  <p>This is my code for getting the full html page:</p>
  
  <pre><code>Document doc = Jsoup.connect("http://www.4shared.com/get/i-EbooI0/batman_hd.html").timeout(0).get();
  System.out.println(doc.toString() + "\n\n\n\n");
  Elements links = doc.select("input[type=hidden]");
  for (org.jsoup.nodes.Element link : links) {
      System.out.println(link);
  }
  </code></pre>
  
  <p>View Screenshot of needed values here </p>
  
  <p><a href="https://i.stack.imgur.com/HUHVx.jpg" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/HUHVx.jpg" alt="enter image description here"></a></p>
  
  <p><strong>SOLUTION</strong></p>
  
  <pre><code>Connection.Response response = Jsoup.connect("myUrl")
      .method(Connection.Method.GET)
      .execute();
  
  Document homePage = Jsoup.connect("myUrl")
      .cookies(response.cookies())
      .get();
  </code></pre>
  
  <p>Modified version of code described here: <a href="https://stackoverflow.com/questions/7139178/jsoup-cookies-for-https-scraping">Jsoup Cookies for HTTPS scraping</a>. This gets the cookies as suggested by Niranjan and then reconnects to your Url.</p>