Hackernews thread requesting for a search engine that excludes the world's biggest websites
https://news.ycombinator.com/item?id=23202850
Ask HN: Is there a search engine which excludes the world's biggest websites?
2020-05-16
None
<div><div><td><table>
<tbody><tr>
<td></td> <td></td><td></td></tr><tr><td></td><td>
</td></tr>
<tr></tr><tr><td></td><td><div><p>Discovering unknown paths of the web seems almost impossible with google et al..</p><p>Are there any earch engines which exclude or at least penalize results from, say, top 500 websites?</p></div></td></tr> </tbody></table><table>
<tbody><tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><br/><div>
<p></p></div></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td><table> <tbody><tr> <td></td><td>
 </td><td><p></p></td></tr>
</tbody></table></td></tr>
<tr><td></td></tr>
</tbody></table>
</td></div></div>||||I|||| Hacker News new | past | comments | ask | show | jobs | submit login          
           

  Ask HN: Is there a search engine which excludes the world's biggest websites?         
 578 points by cJ0th on May 16, 2020 | hide | past | favorite | 228 comments          
           
 Discovering unknown paths of the web seems almost impossible with google et al..          
           
 Are there any earch engines which exclude or at least penalize results from, say, top 500 websites?          
           
           
           
           
 noad on May 16, 2020 | next [–]           
           
 This is a great question, I also want a way to search the internet but exclude all major media domains as well as any company over a certain size. So I just want to search through old blogs, SO, non-corporate social media, weird forums, etc.        
           
 There are so many cool things I remember reading on the web like 10-20 years ago that still exist that are so buried now on Google they might as well not exist. Nowadays searching any topic seems to always lead you to CNN and Microsoft and Facebook and other huge corporations. Search results are just becoming more sanitized and beige and meaningless every day.      
           
           
 dorkwood on May 16, 2020 | parent | next [–]           
           
 For years, my trick for finding interesting content was to go to, say, the 7th page of Google results, and start there. This doesn't work anymore -- it's SEO-optimised listicle blog posts all the way down.        
           
 My trick now is to use Twitter to discover interesting people, and follow them there. Granted, it's not a search engine, but it's at least given me the ability to discover weird things again.        
           
           
 gbin on May 17, 2020 | root | parent | next [–]           
           
 I agree on the original issue but social media tailored to people's own bubble is probably not a good source for enlightenment either...         
           
           
 dorkwood on May 17, 2020 | root | parent | next [–]           
           
 I think it depends how you use it.           
           
 One of the things I enjoy doing on Twitter is posting up something I'm working on, and then clicking through to all the profiles of the people who like, comment, or retweet my work. I stumble across an incredibly diverse range of people by doing this, many with conflicting opinions to my own, and many who belong to strange subcultures that I don't understand, but who were all drawn to my work for one reason or another.     
           
 I think there's definitely a danger of crafting a bubble for yourself if you choose to use it that way, but as a tool for discovering people making cool stuff who otherwise wouldn't cut through the noise on something like Google search, I haven't found anything better.       
           
           
 tomaskafka on May 17, 2020 | root | parent | prev | next [–]          
           
 Many times I have thought about what could happen if Twitter asked you to recommend up to 3-5 people you value, and write a tweet-sized (or shorted) recommendation.         
           
 Over time you would get a 'pagerank for people' and could do awesome stuff with that, like 'You don't know XYZ, but 3 people you trust trust her, and this is what they tell about her:' ...        
           
           
 halfdan on May 17, 2020 | root | parent | next [–]           
           
 Until that gets screwed using social engineering and people optimising for being affiliated with people writing them good reviews.         
           
           
 Terretta on May 17, 2020 | root | parent | next [–]           
           
 You know, for people with Klout.           
           
 https://en.wikipedia.org/wiki/Klout           
           
           
 edeion on May 18, 2020 | root | parent | prev | next [–]          
           
 I was about to ask about dmoz.org. But apparently it's dead. We could probably do something with bookmark sharing à la Delicious. Good dead things for a better future.         
           
           
 Scoundreller on May 16, 2020 | parent | prev | next [–]          
           
 Heh, I was trying to do research on coronaviruses (of which COVID-19 is one of many coronaviruses), but Google sanitized the result and only showed me "official" COVID-19 resources and buried the broader coronavirus resources.        
           
 https://www.google.com/search?q=coronavirus           
           
           
 stolenmerch on May 16, 2020 | root | parent | next [–]           
           
 COVID-19 isn't a coronavirus, it's the disease caused by the SARS-CoV-2 virus.          
           
           
 ganstyles on May 16, 2020 | root | parent | next [–]           
           
 Giving you the benefit of the doubt, and assuming this isn't just pedantry, especially since you're getting downvotes (because I assume everyone thinks this is just pedantic correction) I looked it up.        
           
 In the context of "trying to do research on coronaviruses" your comment appears to be not only correct but an important distinction, rather than the pedantry it appears to be.         
           
 From Wikipedia: "...more lethal varieties [of coronaviruses] can cause SARS, MERS, and COVID-19."          
           
 And...           
           
 "Severe acute respiratory syndrome coronavirus 2 [SARS-CoV-2] is the strain of coronavirus..."          
           
 I learned something today!           
           
           
 ccheney on May 16, 2020 | root | parent | next [–]           
           
 I also had this epiphany awhile back and what helped me understand the difference was to think about how HIV (the virus) causes AIDS (the disease).         
           
           
 teamspirit on May 16, 2020 | root | parent | prev | next [–]          
           
 Further, CoViD-19 literally stands for: Co[rona] Vi[rus] D[isease] - [discovered in 20]19 - "D" standing for disease caused by this particular strain.         
           
 To be honest, I was a bit disappointed when I found out, though I admit now it's a little refreshing to have be so simply named.         
           
           
 Theodores on May 17, 2020 | root | parent | next [–]           
           
 Which can be further abbreviated as C19. I have seen this in personal chats and wonder how long it will be before it gets into newspaper headlines where space is at a premium in print editions.        
           
           
 Wistar on May 17, 2020 | root | parent | next [–]           
           
 I already see it in publications but more frequently see the variant C-19.          
           
           
 jkoudys on May 16, 2020 | root | parent | prev | next [–]          
           
 Complaining about pedantry is the new pedantry.           
           
           
 acituan on May 16, 2020 | root | parent | prev | next [–]          
           
 I know you are relaying the public information accurately, but I wish authorities pushed better names. Like calling the virus "the virus that we know has a corona and causes these symptoms" and the disease "the disease caused by this virus that has a corona and that causes these symptoms" is circular. Also, it is not true that it is entirely a respiratory syndrome . There are serious non-respiratory symptoms, extent of which we are to discover. Finally, if it is a syndrome causing virus, by definition we wouldn't have the crisp boundaries of a disease around it, which indeed we don't.  
           
 If these were names for services and classes that came in a code review, how many would really approve?          
           
           
 pwdisswordfish2 on May 16, 2020 | root | parent | prev | next [–]          
           
 American media calls both disease and virus "corona". They seem to care little about such details.          
           
           
 perl4ever on May 17, 2020 | root | parent | next [–]           
           
 I've been using "covid". Don't know if it will catch on, but I feel it's important for efficiency's sake to try to save one letter.         
           
           
 docbrown on May 16, 2020 | root | parent | prev | next [–]          
           
 [retracted] and I hope this is just a misunderstanding. As the director of the World Health Organization (WHO) said, 2019-nCoV is a novel (new) coronavirus.[0] The CDC defines coronavirus as a virus that was not previously known — check the FAQ, “what is a novel coronavirus?”[0.5]       
           
 They changed the name of this coronavirus to reflect the disease more accurately to COVID-19.[1]          
           
 The CDC has a list of other coronavirus’ that have existed.[2]          
           
 0: https://twitter.com/DrTedros/status/1227297754499764230?s=20          
           
 0.5: https://www.cdc.gov/coronavirus/2019-ncov/faq.html          
           
 1: https://www.who.int/dg/speeches/detail/who-director-general-...          
           
 2: https://www.cdc.gov/coronavirus/types.html           
           
 ——           
           
 Edit: Since there seems to be a misunderstanding from everybody’s part on this as it’s referred to as both and often interchangeably in a mainstream setting, take a look at John Hopkins guide: https://www.hopkinsguides.com/hopkins/view/Johns_Hopkins_ABX...       
           
           
 okintheory on May 16, 2020 | root | parent | next [–]           
           
 No, the comment above is correct, and your own links show this.          
           
 From link [2]: "SARS-CoV-2 (the novel coronavirus that causes coronavirus disease 2019, or COVID-19)"          
           
 The previous comment was just making the point that the (new) virus is called SARS-CoV-2 and the associated disease is called COVID-19.         
           
           
 COVIDisntCORONA on May 16, 2020 | root | parent | prev | next [–]          
           
 Excuse the incivility, but no.           
           
 COVID-19: disease caused by SARS-CoV-2           
           
 SARS-CoV-2: strain of SARS-CoV           
           
 SARS-CoV: severe accute respiratory syndrome coronavirus          
           
 Coronavirus: virus that causes respiratory diseases in mammals, such as SARS (SARS-CoV) MERS (MERS-CoV), and COVID-19 (SARS-CoV-2)         
           
           
 tacon on May 16, 2020 | root | parent | next [–]           
           
 >SARS-CoV-2: strain of SARS-CoV           
           
 Excuse the incivility, but no. SARS-CoV-2 is not a strain or type of SARS-CoV. The viruses share ancestors, but SARS-CoV-2 did not come directly from SARS-CoV. SARS-CoV and SARS-CoV-2 are in the category of beta coronaviruses[0].        
           
 "The whole genome-based phylogenetic analysis presented that two Bat SARS-like CoVs (ZXC21 and ZC45) were the closest relatives of SARS-CoV-2."[1]         
           
 [0] https://www.mdpi.com/2076-0817/9/3/240/htm           
           
 [1] https://www.mdpi.com/pathogens/pathogens-09-00240/article_de...          
           
           
 COVIDisntCORONA on May 16, 2020 | root | parent | next [–]          
           
 Excuse the incivility once again, but no.           
           
 While we're on the topic of linguistic pedantary, strain isn't exclusive to direct mutations from a parent genome. Strains, like much of biological taxonomy, are a human abstraction to make communication of the idea of -- in this case -- "a virus sharing similar properties to coronaviruses that cause severe acute respiratory syndrome" -- albeit this is a very simplified definition for the sake of brevity.     
           
 SARS is caused by SARS-CoV-1 and COVID-19 is caused by SARS-CoV-2.          
           
 Rather, if we would like to be absolutely correct about these classifications, we would say SARS-CoV-1 and SARS-CoV-2 are both strains of SARSr-CoV (Severe accute respiratory syndrome related coronavirus), which in itself is a species, an abstract concept used to group related organisms into a convenient umbrella term.      
           
 There is no "eukaryote" organism the same way there is no "SARSr-CoV" organism. The added "r" was a recent addition when COVID-19 was discovered.         
           
 I will cede that I didn't specify this last point, and you were correct to point it out.          
           
           
 egberts1 on May 17, 2020 | root | parent | next [–]           
           
 Wow. A civil exchange on proper use of terms by two disparate but related technological domain experts. It’ll go on forever.         
           
           
 tacon on May 16, 2020 | root | parent | prev | next [–]          
           
 >we would say SARS-CoV-1 and SARS-CoV-2 are both strains of SARSr-CoV          
           
 Thank you for making my point, again.           
           
           
 COVIDisntCORONA on May 16, 2020 | root | parent | next [–]          
           
 I don't believe I did.           
           
 Genera -- as in SARS-CoV-2's genus is Betacoronovirus -- don't have "strains."          
           
 Only families -- such as the SARSr-CoV family -- have strains.          
           
           
 nulbyte on May 17, 2020 | root | parent | next [–]           
           
 To GP's point, your initial reply said:           
           
 > SARS-CoV-2: strain of SARS-CoV           
           
 GP was pointing out that this was incorrect, and you just made that point by stating it yourself.          
           
 Assuming you are intending to engage in the conversation and not be a pedant, I might let you know that your replies are coming across quite coarsely. More specifically, as to prefaces on earlier comments, there is no need to excuse incivility, because there is no need for incivility here.       
           
           
 aSplash0fDerp on May 17, 2020 | root | parent | next [–]          
           
 I found the exchange to be more than civil, with pleasantries not being taken in the literal sense.          
           
 At least this did not fall into the category of "Cold regurgitation of data" (quite popular it seems) and had a level of warmth that was an indication of passion, more than anger (from all parties).        
           
 If they added a temperature social cue to HN comments..... That would be funny.          
           
           
 ArnoVW on May 16, 2020 | root | parent | prev | next [–]          
           
 I'll add that the family of viruses is called Corona virus because they are 'crown shaped'          
           
           
 amelius on May 16, 2020 | root | parent | prev | next [–]          
           
 Kind of strange that the name of the virus contains the symptoms of the disease (SARS = severe accute respiratory syndrome) while the disease doesn't.         
           
           
 rczhang on May 16, 2020 | root | parent | prev | next [–]          
           
 You can just search for 'Coronaviridae' instead. The overwhelming majority of people searching for coronavirus want pandemic news.         
           
           
 pwdisswordfish2 on May 16, 2020 | root | parent | prev | next [–]          
           
 "Is there a search engine which excludes the world's biggest websites?"          
           
 There was "rebranded" web search that someone created a number of years ago and posted on HN that aimed to exclude the top websites from results. I cannot remember the name he gave to the project.        
           
 One way to exclude the world's biggest websites when using Google is to restrict the search to TLDs other than .com, .net and .org. The root zone is full of silly new TLDs that no one uses for large websites. There are hundreds to choose from.        
           
 https://www.google.com/search?q=coronavirus+site:edu           
           
 Looks like Google Scholar is including a number of "coronavirus links" on the main page but thankfully not in the results.         
           
 https://scholar.google.com           
           
 https://scholar.google.com/search?q=coronavirus           
           
 Why not skip Google and "web search" and use a database that does not include all the crap one finds on the www          
           
 Something like           
           
 https://pubmed.ncbi.nlm.nih.gov?term=coronavirus           
           
 or           
           
 https://search.crossref.org/?q=coronavirus           
           
           
 ikeyany on May 17, 2020 | root | parent | prev | next [–]          
           
 In that case, I would omit webpages from the past year.          
           
           
 drran on May 16, 2020 | root | parent | prev | next [–]          
           
 Try http://biomed-sanity.com/ .           
           
           
 nojito on May 17, 2020 | root | parent | prev | next [–]          
           
 Why would Google showcase non official sources?           
           
           
 40four on May 16, 2020 | root | parent | prev | next [–]          
           
 It is kind of funny that we talk about SARS-COV-2 as if it is the only coronavirus. Coronavirus, singular. If I’m not mistaken the common cold is in the corona virus family.         
           
           
 computerfriend on May 16, 2020 | root | parent | next [–]          
           
 The common cold is caused by coronaviruses and rhinoviruses. The skew is heavily towards rhinoviruses.          
           
           
 40four on May 16, 2020 | root | parent | next [–]           
           
 Rhinovirus, that’s right. Thanks. I saw an article recently about colds caused by coronaviruses so it was fresh in my head.         
           
           
 chris_f on May 16, 2020 | parent | prev | next [–]           
           
 If you are looking for a forum search, check out https://boardreader.com/ .          
           
 I have a theory that web crawling alone is not the best way forward to find the most relevant results because of the volume of content continually being created, much of which is niche and sometimes dynamic.        
           
 Instead I believe linking together vertical search sources that have targeted information based on search intent will provide better results.         
           
 I created Runnaroo [0] for that purpose. If you search a programing question, it will pull traditional organic results from Google, but it will also directly query Stack Overflow for a deeper search.        
           
 [0] https://www.runnaroo.com           
           
           
 scottlocklin on May 17, 2020 | root | parent | next [–]          
           
 This is really good!           
           
 https://www.runnaroo.com/search?term=J+apply+verb           
           
 versus google's completely useless:           
           
 https://www.google.com/search?safe=active&q=J+apply+verb&oq=...          
           
 Everyone else, get in here: this is top notch stuff.           
           
           
 lazyjones on May 16, 2020 | parent | prev | next [–]           
           
 This is somewhat ironic because 20 years ago, hobbyists would frequently put their obscure personal pages on Geocities and other large corporation's web space.         
           
           
 wolco on May 16, 2020 | root | parent | next [–]           
           
 Memory can be foggy but the most useful were hosted on university pages or random folders off a random domains or you get a subdomain. I picked the username 'search' which gave me search.batcave.net which worked great until one day they just took over the subdomain for a site wide search. They were confused when I complained.      
           
 Sure people hosted on geocities and tripod and they were the biggest and easiest to remember. But quality of a geocities page compared to a mit student page was much lower.         
           
           
 CalRobert on May 16, 2020 | root | parent | prev | next [–]          
           
 20 years ago, it was extremely common for your ISP to give you 5MB or whatever of space to use. users.ispname.com/~yourname or whatever. It was great, tbh, since anyone with cuteftp and notepad could publish to the world.        
           
           
 ObsoleteNerd on May 16, 2020 | root | parent | next [–]          
           
 Still common now, at least in Australia. My ISP gives me 1GB if hosting for a website and 10 mailboxes.          
           
           
 fedede on May 17, 2020 | parent | prev | next [–]           
           
 Hey! I actually liked this idea and I'm considering starting a learning project on it. I've seen a lot of interest and ideas in the comments, and decided to create a very short Google form to start gathering all the interested people so we can organise something interesing. Is anybody in? :)       
           
 https://forms.gle/5KuTYVdYaMzRD2n78           
           
           
 ryandrake on May 17, 2020 | parent | prev | next [–]           
           
 If there was a way to simply exclude from searches: shopping, news, images, videos, and “listicles”, I think that would get us most of the way.         
           
 Especially shopping. The endless stores are the worst part of search results. If I search for anything that remotely looks like a product, the results are just choked with store after store trying to sell me the thing. Awful.        
           
           
 type0 on May 17, 2020 | root | parent | next [–]           
           
 If you do your own search engine like that, implementing rules that block affiliated commerce blogs from appearing would help alot.         
           
           
 sanqui on May 16, 2020 | prev | next [–]           
           
 There is a search engine with this exact goal: https://millionshort.com/ .          
           
 I haven't had that great results with it myself though.          
           
           
 smackay on May 16, 2020 | parent | next [–]           
           
 I tried it with "waders" which are either the things that you to put on your feet to go fishing or a category of birds (shorebirds or herons). The results after going for all the options were still exclusively stores wanting to sell me the former.       
           
 Garbage in, garbage out. I guess. Still I like the idea of something to side-step the SEO perhaps with more effort they can make it work but relying on Google or any major search engine for the base results is the wrong way to go.        
           
           
 glenstein on May 16, 2020 | root | parent | next [–]           
           
 I tested your search term and had similar experience. I have, however, had positive experience with other categories, such as philosophy. Searching Wittgenstein with the top million sites removed, I found some gems: a play, a disney character that was a supercomputer named after Wittgenstein on a direct to video movie which I learned was later partly inspiration for Wall-E, Wittgenstein-oriented societies, awards, and general philosophy references I had never heard of.    
           
 I suppose it depends on the category.           
           
           
 spaceman_2020 on May 16, 2020 | root | parent | prev | next [–]          
           
 I've found that you get far better results for many queries on YouTube now since it's tougher to mass spam YouTube with low quality content (compared to churning out a $5 article).         
           
           
 Nevada-Smith on May 16, 2020 | root | parent | prev | next [–]          
           
 Google Scholar "waders" [1]           
           
 [1] https://scholar.google.com/scholar?hl=en&as_sdt=0%2C29&q=wad...          
           
           
 behnamoh on May 16, 2020 | parent | prev | next [–]           
           
 Man it's super fast! I wish Google were like this.           
           
           
 webspinner on May 19, 2020 | parent | prev | next [–]           
           
 Oh, I've had epic results with Millionshort!! I mean, as far as stripping most everything away, and just surfing.          
           
           
 erikbye on May 16, 2020 | prev | next [–]           
           
 For google you can use this https://addons.mozilla.org/en-US/firefox/addon/g-search-filt... , just drop in your list of those 500 URLs, once you've decided on what the top 500 is.         
           
 For other engines you can use https://addons.mozilla.org/en-US/firefox/addon/greasemonkey/ with this script https://greasyfork.org/en/scripts/1682-google-hit-hider-by-d...         
           
           
 atrudeau on May 17, 2020 | parent | next [–]           
           
 For Chrome: https://chrome.google.com/webstore/detail/google-search-filt...          
           
           
 terrycody on May 17, 2020 | parent | prev | next [–]           
           
 really nice, thx for share!           
           
           
 thekyle on May 16, 2020 | prev | next [–]           
           
 There is Million Short which allows you to search without the top 100, 1k, 10k, 100k, or 1m sites. Personally what I'd like to see is a search engine that only indexes webpages without ads since that should eliminate lots of the SEOd garbage. It would also be nice to use the text to code ratio to derank JS heavy sites.      
           
 https://millionshort.com/           
           
           
 b0ner_t0ner on May 17, 2020 | parent | next [–]           
           
 > like to see is a search engine that only indexes webpages without ads since that should eliminate lots of the SEOd garbage         
           
 Not just ads, but also ranked by the number of third-party cookies/tracking scripts a site has.          
           
           
 joshspankit on May 17, 2020 | parent | prev | next [–]           
           
 And blacklist anything with those outbrain blocks. “One weird trick to _____.”          
           
 Very surprised where I see those these days, and they always make me run away.          
           
           
 dorkwood on May 17, 2020 | root | parent | next [–]           
           
 The fact that news websites happily adopted those is disgusting. They're literally tricking people into thinking it's news content.         
           
           
 icedistilled on May 17, 2020 | root | parent | next [–]          
           
 And they also, for the longest time, strongly turned me off paying for any news sites. The more they use scummy ads that turn people off, the more they need ad revenue.         
           
 That's a nice negative feedback loop or catch-22           
           
           
 tlarkworthy on May 16, 2020 | prev | next [–]           
           
 I made a script on ObservableHQ to surf YouTube psuedo-randomly https://observablehq.com/@tomlarkworthy/random-place-on-yout...         
           
 I do a random city + documentary as the search term, it's taken me all over the world and seen some very strange things.         
           
 One of my favourites was Aarhus, which had a Danish language rapper proclaiming he was putting Aarhus on the global map (I have never heard of the city of Aarhus). https://youtu.be/WSZxuzgImLo They dis Copenhagen a lot too, lol. You get a more intimate YouTube experience with the low view videos       
           
 But I also seen amazing religious rituals. An excellent documentary on Karachi.          
           
 Because it's observable hq you can fork it and figure out your own algorithm for biasing the random.          
           
           
 totemandtoken on May 16, 2020 | prev | next [–]           
           
 Reminds me of this classic pg essay: http://www.paulgraham.com/ambitious.html          
           
 Specifically this quote: "The way to win here is to build the search engine all the hackers use. A search engine whose users consisted of the top 10,000 hackers and no one else would be in a very powerful position despite its small size, just as Google was when it was that search engine."       
           
 There has been a lot of grumblings about the state of search these days. Maybe the time is nigh for a new search engine?         
           
           
 tetris11 on May 16, 2020 | parent | next [–]           
           
 I feel that we should go down the adblock hosts list approach, where people download website lists from individuals they trust who have curated or scraped links of websites complete with keywords, and its up to the user to refresh their lists and then perform a search on their website.txt file       
           
 It will be limited, but still quite powerful, similar to the way that we can pick and choose different host file sources from the web.         
           
           
 rthomas6 on May 17, 2020 | root | parent | next [–]           
           
 Does anyone else remember StumbleUpon? It's not exactly the same as a search engine, but that worked really well for finding interesting content back in the day.         
           
           
 ublaze on May 17, 2020 | parent | prev | next [–]           
           
 I've wondered if this can be built with a limited budget, in terms of cloud costs. At least it feels like you'd need a lot of hosts to satisfactorily index the web, and store the results so that you can return results instantaneously.        
           
           
 joshspankit on May 17, 2020 | root | parent | next [–]           
           
 With limited budget and the right results, I think people would pay with “non-instant” results. Even 5-10 seconds might be perfectly fine, and that’s “easy” on a limited budget.         
           
           
 igammarays on May 16, 2020 | prev | next [–]           
           
 DEVONagent is a highly configurable search utility which can be used to combine and de-duplicate results from multiple search engines at once, exclude sites or keywords from a blacklist, follow deep links within search pages, and perform some filtering logic on the text of results.       
           
 Before I knew about DEVONagent I would often just search multiple engines and sources trying to find something particular (e.g. a particular PDF) or unique results.         
           
 https://www.devontechnologies.com/apps/devonagent           
           
           
 lemonberry on May 16, 2020 | parent | next [–]           
           
 Thank you for the link. This looks really cool. I used DEVONthink years ago. It seemed like a great piece of software but I didn't have a great use case for it. Looking forward to checking out DEVONagent.        
           
           
 petra on May 16, 2020 | parent | prev | next [–]           
           
 It looks really interesting. Sadly, it's Mac only.           
           
 Does anybody knows of something similar for Windows or Linux ?          
           
           
 brightball on May 16, 2020 | parent | prev | next [–]           
           
 This does look really interesting. Thank you.           
           
           
 pavelmark on May 16, 2020 | prev | next [–]           
           
 Simply removing Pinterest would be a huge step in the right direction.          
           
           
 joshuaissac on May 16, 2020 | parent | next [–]           
           
 I use an add-on called Unpinterested! to remove Pinterest results from my Google search results:          
           
 https://github.com/sellomkantjwa/unpinterested           
           
 All it does is add -site:pinterest.com to the search bar for image results (can be configured to also do it for Web results), but it gets the job done.         
           
           
 pier25 on May 16, 2020 | parent | prev | next [–]           
           
 And quora           
           
           
 stock_toaster on May 16, 2020 | root | parent | next [–]          
           
 ye gods, yes.           
           
           
 user9429450 on May 16, 2020 | parent | prev | next [–]           
           
 Any insight with regard to how Pinterest was able to do what they've done? Did they simply dump more money into it?          
           
           
 chaos_a on May 16, 2020 | prev | next [–]           
           
 https://wiby.me/ exists to solve this exact problem. I've found some pretty neat/odd websites on it in the past.          
           
           
 lucb1e on May 16, 2020 | parent | next [–]           
           
 The "About" information is, counter-intuitively, under "Settings":          
           
 > In the early days of the web, pages were made primarily by hobbyists, academics, and computer savvy people about subjects they were interested in. Later on, the web became saturated with commercial pages that overcrowded everything else. All the personalized websites are hidden among a pile of commercial pages. [...] The Wiby search engine is building a web of pages as it was in the earlier days of the internet.     
           
           
 btrettel on May 16, 2020 | root | parent | next [–]           
           
 That makes me think: Is there a search engine which removes pages with any ads or affiliate links? That might be the easiest way to remove the commercial pages.         
           
           
 lucb1e on May 16, 2020 | root | parent | next [–]           
           
 I was just thinking pages with external dependencies, in the spirit of the old web, but your idea sounds a lot more reasonable. Not sure if that exists, I'd be interested!         
           
           
 generalpass on May 16, 2020 | parent | prev | next [–]           
           
 It is a carefully curated directory, which is problematic.          
           
 For example, I submitted Pizza Hut's archived original web page [1], but it wasn't added.          
           
 Even for a search engine exposing niches, updating a directory manually will likely be too slow, unless the directory is maintaining a single nich (e.g., unladen airspeed of every species of swallow), but then we end up with some insane number of search engines and how to select which one?       
           
 [1] http://www.pizzahut.com/assets/pizzanet/home.html           
           
           
 kd5bjo on May 16, 2020 | root | parent | next [–]           
           
 Especially if you’re focussing on evergreen information, there’s no reason why people can’t have their own personalized crawler and index— I’ve occasionally thought about rolling my own with a browser extension that lets me add seeds at the click of a button.       
           
           
 zxexz on May 16, 2020 | root | parent | next [–]           
           
 I've been working on something like this for my own use - I'm not a fan of browser-based history. My home-rolled solution is starting to be good enough where I can use it to easily find exactly what I'm looking for, assuming I've previously read it, by both searching the title and URL, as well as the content on that page (my major gripe with "History" in Chrome and Firefox is that it doesn't search the page content, and if it did, syncing it would have major privacy concerns).    
           
 The problem I'm running into is that I still have to use major search engines to find new content, way more than I'd like. I hope to make my local service available open source once I have 'federated' history search working, so that we can have a primitive search engine and share with people we trust. Also need to work out some security issues - it's scary having all the content you read and see on your home network, protected only by your hackily-patched-together security.    
           
 EDIT: Actually I'd like to elaborate a bit more in case anybody actually reads this and has any ideas. On the desktop side, it's pretty easy. Initially started out MITMing my own traffic with a self-signed cert added as a root cert to all my machines. This only works on my home network, so I did a VPN thing. This was way to clunky and the security concerns are innumerable. I ended up biting the bullet and writing a chrome extension which works wonderfully, except for some slight performance issues.   
           
 However, I wish to also archive my phone content - I read just as much on my phone as my computer. I can do it on Android with the MITM process, but the same issues as above still apply, and it doesn't work with iOS (at least I can't find a way).        
           
 I'm thinking of taking an open source project, like Firefox/Fennec and building it in to the app itself. In that case it may make sense to forgo the browser extension and just roll my own forked browser on every platform, even iOS. I don't know much about iOS dev though.       
           
           
 ehonda on May 16, 2020 | root | parent | prev | next [–]          
           
 I clicked your link, but I don't see an archive, its redirecting me to their main website.          
           
 Wiby is based around two main things:           
           
 Non commercial content (1) that does not rely heavily on excessive javascript and CSS (2).          
           
 http://wiby.me/submit contains the submission criteria.          
           
           
 bsanr2 on May 16, 2020 | parent | prev | next [–]           
           
 I tried this yesterday. It seems biased towards the interests of the curators, and, like Google in regards to "some ideal average consumer", is therefore useless if you fall outside a certain level of similarity to the target demo.        
           
 For example, I enjoy weightlifting and strength sports. I did a search for "muscle", and every result but one was using the word "muscle" as a figurative metaphor. Barely anything about actual muscles. Searching "funk" was just as bad. One page about Motown and a LOT of midis.       
           
           
 keenmaster on May 16, 2020 | root | parent | next [–]           
           
 What if Google Advanced Search produced a visual network map which showed you the salient clusters of terms related to your search? You’d then be able to click on a cluster and the search results would change to adjust to what you’re really searching for.       
           
 Ex: The network map for “weightlifting” would include many clusters, but 2 big ones would be the hypertrophic cluster (surrounded by a bunch of related terms) and toning cluster (calisthenics would be under this cluster for example). Click on either and the results will change accordingly.       
           
 This would actually work even better for subjects you don’t know much about, because Google will teach you about the salient clusters in that field. The clusters could be enhanced with popular images associated with each term. Popular clusters would display as larger than others.       
           
           
 miek on May 17, 2020 | root | parent | next [–]           
           
 I like this idea a lot. Wonder if anyone is working on this.          
           
           
 keenmaster on May 17, 2020 | root | parent | next [–]           
           
 No one that I’m aware of. I wouldn’t mind if Google hired me to help make it a reality. I have an email address in my bio (lol).         
           
           
 ehonda on May 16, 2020 | root | parent | prev | next [–]          
           
 Hi, just wanted to clear something up. Wiby is biased towards the interests of those who submit websites to it.          
           
           
 EmilioMartinez on May 17, 2020 | parent | prev | next [–]          
           
 Just added wiby.me/surprise to the bookmarks bar. Amusingly, the icon keeps changing every time I use it.          
           
           
 Aeolun on May 16, 2020 | parent | prev | next [–]           
           
 But to add my site I have to add every page individually. Nobody ain’t gonna use thst.          
           
 I’d have to submit every blog post?           
           
           
 ngold on May 17, 2020 | parent | prev | next [–]           
           
 Can't wait to try this search engine. Thank's for the link.          
           
           
 mikekchar on May 17, 2020 | prev | next [–]           
           
 Here is an idea that I've always wanted to do, but will never have time for: A curated search engine.          
           
 Basically the idea is to have people band together and "recommend" links. You then do your normal spidering of the websites to create a search engine (or even just call through to a number of existing search engines). However, the ranking of the results is based on the weighting of the recommendations.       
           
 It's essentially a white list based on your own personal bubble. Of course this won't work in general because you will always get SEO creeps spamming recommendations. However, it gives you tools for working around those creeps. The average person probably won't be able to manage it, but power users probably will.      
           
 By not trying to solve the problem for everybody, it makes it easier to solve to problem for some people. Or at least that's my thesis :-) I might be wrong.         
           
           
 netsectoday on May 17, 2020 | prev | next [–]           
           
 You can boot up your own custom search engine in a few minutes with YaCy (Ya See!) an open-source, P2P, Dockerized crawler and search engine built on top of Solr.         
           
 https://yacy.net/           
           
 If you're generous; you can make your index available to other P2P instances.          
           
 I wanted to run an API search the other week and was blown away with how quickly I could prop-up my own custom search portal (I didn't want to pay for API access to other search engines, and YaCy comes with a JSON and Solr endpoints).        
           
 I ran it locally to test my crawl filters, then pushed a private instance out to Digital Ocean to turn up the heat with the crawling. The only issue I had was the crawler would hit the max memory threshold on long crawls and the container would restart, but that was fixed by scaling up the box.       
           
           
 l72 on May 18, 2020 | parent | next [–]           
           
 I have my own yacy search engines running internally (non-peered) for similar reasons. One crawls some key code documentation sites that I need for work, and another crawls a whole bunch of music blogs.        
           
 While I typically still use RSS for reading music blogs, I find having the search engine is a great way to go back and find something or discover something new! Every time I find a new blog, I just add it as an index to yacy to crawl.        
           
 I think it'd be great to see people spinning up larger instances that are highly specialized. For example, maybe a search engine that is dedicated solely to sci-fi and only crawls high quality boards, personal sites and blogs, and skips all the spammy, seo-optimized sites.       
           
           
 crawlcrawler on May 16, 2020 | prev | next [–]           
           
 I built a search engine for this and other, similar purposes. With Crawl Crawler you start out by searching the meta data of a Common Crawl ("CC") crawl. Then you define a sub section of that data collection by designing a query which search result includes your favorite sites. Then you enrich that sub section by linking those meta data documents (that come from CC's WAT repo) to full text extracts or HTML from CC's WET repo or the WWW. Then you set it to recurringly refresh that section. Voila! You have created a search index that includes your preferred sites. https://crawlcrawler.com  
           
           
 chris_f on May 16, 2020 | parent | next [–]           
           
 This is pretty cool. I always wondered why there wasn't a user interface search somewhere for the CommonCrawl data.          
           
           
 allwynpfr on May 16, 2020 | prev | next [–]           
           
 You should try million short. As the name suggests, it takes our the first 100 / 1k / or a million results so you're left with those that aren't all that popular. That seems to be what you're looking for. https://millionshort.com/        
           
           
 nic-waller on May 16, 2020 | prev | next [–]           
           
 My hobby project is https://random.surf (works better on desktop than mobile).          
           
 I share that same desire to visit the web less travelled. I want to discover interesting sites that deserve to be bookmarked because they will never show up in a search engine.         
           
           
 77ko on May 17, 2020 | parent | next [–]           
           
 Love it! Discovered something interesting very quickly. Bookmarked for future use.          
           
           
 webspinner on May 19, 2020 | parent | prev | next [–]           
           
 That's definitely spinning the web, which is a whole lot of fun.          
           
           
 severine on May 17, 2020 | parent | prev | next [–]           
           
 Thank you!           
           
 True "Interdimensional Cable" vibes.           
           
           
 tetris11 on May 16, 2020 | parent | prev | next [–]           
           
 This is excellent           
           
           
 dangoljames on May 16, 2020 | prev | next [–]           
           
 There used to be java applet embedded in altavista.com's website that could be run against search results. It would do semantic processing on the results and present a list of generated terms, each with a checkbox. Checking a box would pull any returns which contained the topic from the remaining search results.      
           
 This was fire. If a topic were being discussed on the web, you could find it with this tool. Unfortunately, it did not fit the vision of the parasitic overlords who bred us to produce and consume for their benefit.        
           
           
 visarga on May 16, 2020 | parent | next [–]           
           
 Altavista itself was a junk search engine though, especially after they sold out and the new owner stuffed it with ads.         
           
           
 dennisy on May 16, 2020 | prev | next [–]           
           
 I think you could get good results if you just penalise sites for the number of third party JS. Which shows by proxy a more established site/corp.         
           
 You could add a bunch of heuristics such as size, number of links etc.          
           
 Maybe even train a classifier to select the “smaller” part of the web.          
           
           
 inopinatus on May 16, 2020 | prev | next [–]           
           
 I would pay real subscription money for a search engine that focused on knowledge-oriented results rather than retail and commercial results.         
           
 When I type “shoes”, it would give me: links for the functional and creative history of footwear, the taxonomy of shoes, methods of construction, current and historical footwear industry data, synonyms and antonyms, related terms and professions, the dictionary definition, and similar links related to secondary meanings (such as any protective covering at the base of an object, horseshoes etc). I’d also hope for a comedy link to a biography of Cordwainer Smith.    
           
 What I actually get, which I don’t want at all: pages and pages of shoe shopping.          
           
 The various means to exclude “top X sites” are the roughest possible heuristic in that direction, and throw out the baby with the bathwater (for example, a long-established manufacturer may well have an informational online exhibit)        
           
 Google has essentially failed me in its primary mission. Bing at least has the grace to admit they are here to “connect you to brands”. And sadly, right now, every other option is an also-ran.        
           
 In practice I use DDG, directed by !bangs towards known encyclopaedic or domain-specific sources. I am certain that I’m missing out.         
           
           
 seektable on May 17, 2020 | parent | next [–]           
           
 What you describe sounds like a mix of personal knowledge base that seats on the top of existing search engines, public and maybe even private databases. Major difference is here:         
           
 * when you make a query to this knowledge base, it has a history of your prev searches / preferences (not google)          
           
 * it can propose variants of suggestions on what is your intent in this particular query - and make much more detailed queries (auto include/exclude keywords websites etc) to multiple sources (not only google, maybe anonymously)        
           
 * it can parse results from these sources and re-arrange them (use own rank system) according to the your preferences. In this system, you can explicitly say - I hate that, and I like that, and this will affect the behavior. Yes this is 'information bubble' but it is controlled by you and not by google!       
           
 * finally, this system may work in background and handle 'research' search queries. What I mean here: currently, Google is about instant search - it gives you results in milliseconds, and that's all. It cannot spend much computations for more precise, more intellectual check of content in links from the search results - it cannot do reasoning - and you have to do that by yourself: open links from 1-st page, and close most of them immediately b/c they are not relevant for you, go to 2-nd page and so on. It would be cool if most of this could be automated - with modern natural language processing approaches and old-school prolog-like reasoning this is real and not a fantastic from sci-fi.
           
 My vision that this kind of search assistant cannot be SaaS / closed source. It is about the freedom - and thus this should be open source / self-hosted app that can be deployed on PC or on cloud VM - but hosting should be controlled by end-users, not companies.       
           
 I don't know if something like this ever exist. If not, maybe its time to create it.          
           
           
 atlantique on May 16, 2020 | parent | prev | next [–]           
           
 That sounds like the job of an encyclopedia. Maybe some sort of collaborative encyclopedia where people can edit pages and add references.         
           
           
 inopinatus on May 16, 2020 | root | parent | next [–]           
           
 I know you’re only half joking. In practice many, perhaps most of my DDG queries do end in !w - but there’s a wealth of information that is relevant, interesting, and useful, but wouldn’t be considered encyclopaedic, or that is merely summarised in Wikipedia; in addition, their references are included as supporting citations, very far from a comprehensive index of currently accessible information.     
           
           
 text_exch on May 17, 2020 | prev | next [–]           
           
 I've long wanted to build a search engine of only personal blogs. I am less familiar with the field of information retrieval so I haven't gotten started yet, but it's always been a dream of mine and if anyone is interested please contact me at threemillionthflower [at] the world's largest email provider.       
           
 Discovering unknown parts and blogs on the internet is one of the enduring goals of a newsletter that I run [1], which provides a single link to an interesting article every day, usually by lesser-known authors and blogs across the internet.        
           
 [1] www.thinking-about-things.com           
           
           
 webspinner on May 19, 2020 | parent | next [–]           
           
 I'm now a subscriber.           
           
           
 011-video on May 17, 2020 | prev | next [–]           
           
 You are your best search engine !           
           
 On a daily basis your brain use shortcut to get to the point. Open Firefox (of course) ALT+B. Then add a new bookmark for instance :         
           
 Name : Stack Overflow           
           
 Location : https://stackoverflow.com/search?q=%s           
           
 Tags :           
           
 Keyword : st           
           
 Now if you want to search "javascript timer", just type : st javascript timer          
           
 Add "%s" to all your favorites website search url.           
           
 Example : https://en.wikipedia.org/wiki/%s           
           
 To discover some new website content, apply the same trick to Hacker news, Reddit or any RSS River.          
           
 Voila, bye bye GG.           
           
           
 NateEag on May 16, 2020 | prev | next [–]           
           
 For Google, you can ignore specific sites by adding "-example.com".          
           
 See this example of filtering Stack Overflow out of search results:          
           
 https://www.google.com/search?q=loop+over+array+items+in+jav...          
           
           
 1f60c on May 16, 2020 | parent | next [–]           
           
 More specifically, -site:example.com, although there have been reports of Google breaking this time-honored functionality.         
           
           
 brentis on May 17, 2020 | prev | next [–]           
           
 Imagine if sort results had table filters and sort.           
           
 Popularity, Relevance, Age, Type, etc. type could be blog, forum, site, or video. Or like it used to be.          
           
           
 busymom0 on May 17, 2020 | parent | next [–]           
           
 I have been finding recently that Google has been breaking their existing "sort", "time" filters. Try searching for something with site:reddit.com prefix for example and set the time filter to be lets say "Last year". Google still shows you results from 4-5 years ago.       
           
           
 BostonFern on May 17, 2020 | root | parent | next [–]           
           
 I also discovered that recently. It's gone the way of the verbatim constraint.          
           
 Control is being forfeited to steer users back to more profitable content in order to capitalize on a captive market.          
           
 I wonder if being open about it would be so bad for business, instead of the attempt to manipulate users into enjoying the ratcheting-up of their impotence.         
           
 Now, Youtube truncates search results and loads the recommendation stream instead, long before hits are exhausted.          
           
 At least it's been a while since Silicon Valley was keeping the mythical personalized advertising spiel in active circulation.         
           
           
 sneeuwpopsneeuw on May 16, 2020 | prev | next [–]           
           
 I personally use Google Chrome with the duckduckgo search engine. Duckduckgo is not perfect, very in depth searches (such as gameboy advance memory layout only return junk, while google knows you are searching for a nich) but on your average search it is as good as google, somethimes better because it is more factual and will promote less webstores. When it does not give me what im looking for I can add !g anywhere in the question and the same search is done using google.    
           
 Then I use Violentmonkey an open source js/css injector to inject this user script: https://greasyfork.org/nl/scripts/1682-google-hit-hider-by-d... This will block specific domains for you in google, yahoo, duckduckgo etc. I use this to block domains like Quora, sourceforge, cnet and softonic.       
           
 The nice thing about this script is that you can permaban domain you know are junk and they will completely be removed or you can ban a domain like commercial websites. When you ban something it is not removed from google or duckduckgo but it only shows the title in light gray, Im currently experimenting with this on some mayor webstores so I can not really say if this may help you but It can be a good start.     
           
 (edit) I saw some people say why this was not possible before. Google allowed you to block domains and website a few years ago, but they removed this feature. Duckduckgo never allowed you to do that because that would mean that you will have a cookie that remembers your preferences and that is against there principles.      
           
           
 1f60c on May 16, 2020 | parent | next [–]           
           
 > I can add !g anywhere in the question           
           
 I knew about !bangs, but I didn't know you could put them anywhere in the query (e.g. "hello !g world" searches Google for "hello world"). This is going to save me a lot of time on mobile. Thanks!        
           
           
 wumpus on May 16, 2020 | prev | next [–]           
           
 If the question is "Is there a commercially-viable search engine that supports this feature", then the answer is "probably not".         
           
 Implementing this properly involves having your own search index. And that's pretty expensive.          
           
           
 bamboozled on May 16, 2020 | prev | next [–]           
           
 I think on DDG you can do !mil which excludes the first million top ranking sites.          
           
 Edit: Maybe it’s the first million results? I use it to find obscure things sometimes.          
           
           
 mmsimanga on May 16, 2020 | prev | next [–]           
           
 When researching a topic I have had great success searching HN and reading through the comments. If I want to find alternative software tools for a tool I am using the comments on HN are best. Searching through subreddits also yields better results than Google.       
           
           
 DavidPiper on May 17, 2020 | prev | next [–]           
           
 It feels like there could be a (partial) meta solution here:          
           
 A search engine that returns results whose pages weigh in under a certain size.          
           
 From the comments it seems most of the "cruft" filling up Google results are newer web apps, generally JS-heavy and advertising-heavy, etc.         
           
 If you had a filter for pages with (e.g.) < ABC kb of JS, < XYZ external links (excluding img tags), I feel like there'd be a good chance that the "old" web and the "unknown" web would bubble to the top.        
           
 There are plenty of false positives (particularly for "small" forums build with modern JS apps, etc), but it could be one of many filtering tools to achieve better search results.         
           
           
 ngold on May 17, 2020 | parent | next [–]           
           
 Great idea. Google seems to do nothing but remove search options. At least they still have a time filter. Ddg only does a year old I believe.         
           
           
 turnipla on May 16, 2020 | prev | next [–]           
           
 Google used to let you blacklist websites many moons ago, that would go a long way already.          
           
 Now there are a few extensions that do that, but obviously they only hide the results from each page, so sometimes you will see pages with 2 results, if any at all.         
           
           
 rozab on May 16, 2020 | parent | next [–]           
           
 Would be easy to just inject a negative site clause into the query, e.g. `-site:fandom.com`          
           
           
 chrisfrantz on May 16, 2020 | root | parent | next [–]           
           
 Looks like google limits search query length to 2048 characters. That’s probably enough room to exclude a majority of the biggest names.         
           
           
 dublin on May 16, 2020 | root | parent | prev | next [–]          
           
 It would be nice if there were a way to make the exclusion list de the default for all your queries. For instance, I never want to see results from WikiHow again. Ever. Or the New York Times or any of the other paywalled sites...        
           
           
 kortex on May 16, 2020 | root | parent | next [–]           
           
 unpinterested is an extension which simply adds -site:pinterest to image searches. I don't think it'd be hard to do something similar with a custom list.         
           
           
 methou on May 16, 2020 | prev | next [–]           
           
 I used a Google Custom Search Engine (CSE) to remove results from Softonic and alikes, it works well, but still very Google.         
           
           
 petra on May 16, 2020 | parent | next [–]           
           
 Google CSE is a great idea.Tried it in the past.           
           
 But i find the search is at a much lower quality than Google.          
           
           
 dexen on May 17, 2020 | prev | next [–]           
           
 There is a similar problem where Youtube's recommendations and auto-play are mostly big name brands, to the exclusion of individual reporters, commentators, and other content producers. Since recently, a "De-Mainstream Youtube" plugin[1] is available for Firefox and Chrome, fixing that to some extent.       
           
 --           
           
 [1] https://demainstream.com/           
           
           
 bmd3991 on May 16, 2020 | prev | next [–]           
           
 What I’d like to see is a search that excludes any page with ads, and any page with affiliate links. That alone would get rid of 90% of the garbage         
           
           
 chongli on May 17, 2020 | parent | next [–]           
           
 I've expressed this wish on HN a few times in the past as well. One downside to it that I can think of (off the top of my head) is that there may be a lot of small, personal websites that are hosted on a free blogging platform which injects ads into every page. These websites may be very valuable (and not SEO garbage link farms) but would get blocked nevertheless.      
           
           
 dddddaviddddd on May 16, 2020 | parent | prev | next [–]          
           
 Sort of a server-side, page-level adblocker.           
           
           
 bmd3991 on May 17, 2020 | root | parent | next [–]           
           
 It could make for an interesting Firefox extension I think          
           
           
 peel40 on May 16, 2020 | prev | next [–]           
           
 I think there's a simple google way. Just add `-bigwebsite.com` to your query.          
           
 [search term] -google -youtube -facebook ... -top100website and it should work.          
           
 I found a list of the top 1m alexa websites here:           
           
 http://s3.amazonaws.com/alexa-static/top-1m.csv.zip           
           
 An add-on with that list should do the work.           
           
           
 abraae on May 16, 2020 | parent | next [–]           
           
 That's pretty clunky.           
           
 - there's probably a pretty low limit for size of Google queries, you'll likely hit it quickly          
           
 - you won't be able to search for e.g a story about YouTube censoring some content          
           
           
 peel40 on May 16, 2020 | root | parent | next [–]           
           
 I don't know about the current query size limit but I think it's pretty likely to get hit quickly as you correctly pointed out. But, it's useful to use the wildcard "-site" ex: "-site:bigwebsite.com" for excluding just the site, and not the very word being mentioned.       
           
 ex:           
           
 facebook censorship -site:facebook.com           
           
 https://www.google.com/search?q=facebook+censorship+-site%3A...          
           
           
 Mandatum on May 18, 2020 | root | parent | prev | next [–]          
           
 Filtered out regional sites that Google wouldn't cater my search for (eg Taobao, Baidu, etc).. Only possible with 31 negative sites.         
           
 -site:google.com -site:youtube.com -site:facebook.com -site:jd.com -site:yahoo.com -site:wikipedia.org -site:amazon.com -site:netflix.com -site:reddit.com -site:live.com -site:zoom.us -site:okezone.com -site:alipay.com -site:instagram.com -site:twitch.tv -site:csdn.net -site:blogspot.com -site:microsoft.com -site:bing.com -site:github.com -site:tribunnews.com -site:myshopify.com -site:office.com -site:panda.tv -site:stackoverflow.com -site:ebay.com -site:bongacams.com -site:livejasmin.com -site:babytree.com -site:naver.com -site:apple.com <search query>   
           
           
 coronadisaster on May 16, 2020 | parent | prev | next [–]          
           
 your query would obviously be too long for Google           
           
           
 peel40 on May 18, 2020 | root | parent | next [–]           
           
 yeah, that's sad. Google limits its queries to just 32 words.          
           
           
 bhartzer on May 16, 2020 | prev | next [–]           
           
 There is a custom search engine called Newgle.xyz that only shows results from the 1000 or so new gTLDs (new top level domains).         
           
 It’s custom google search results, but since it’s excluding .com, .net, .org etc then you probably won’t see any of the large sites there.         
           
 It’s also interesting to see which sites have been built in the last few years, as the new gTLDS haven’t been around that long.         
           
           
 rkagerer on May 16, 2020 | prev | next [–]           
           
 I would like one that punishes sites with too much ad to content ratio.          
           
           
 loosetypes on May 16, 2020 | prev | next [–]           
           
 What are folks’ non-commoditized heuristics for finding new things online?          
           
 I was intrigued by how dorkweed’s approach has changed over time, as described in a reply to a sibling comment.          
           
 As general search results get watered down and rotten tomato inflation maybe trends towards reflecting company interests rather than my interest-level, maybe it’s worth re-evaluating the vetting avenues we take as users.        
           
 Here’s mine: for games and shows I’ve recently found myself using quantity of fan-videos on YouTube as a proxy for quality. So far it’s been a decent means to find cult followings for something I otherwise wouldn’t necessarily hear about.        
           
 Obviously this approach has its flaws - and is subject to financial perversions to an extent - but I figure if enough people genuinely want to pay tribute to a work, it might be worth checking out.        
           
           
 bluishgreen on May 16, 2020 | parent | next [–]           
           
 How'd you find the quantity of fan videos?           
           
 Personal trick: I follow reaction video blogs, and if they are reacting to something then it is usually worth watching. But reaction blogs are only for short videos and other short form content.        
           
           
 ChrisMarshallNY on May 16, 2020 | prev | next [–]           
           
 Remember sites like stumbleupon?           
           
 I find that the YouTube sidebar is useful for me to find interesting music. I have eclectic tastes, and Google seems to have figured that out. I don't mind.         
           
 I suspect that it would be possible to create a custom API query to Google that would have a "blacklist."          
           
           
 smsm42 on May 17, 2020 | prev | next [–]           
           
 There's Million Short: https://millionshort.com/           
           
 I think they try to do exactly what you ask, but I haven't used them extensively so don't know how good are they.          
           
           
 abarrettwilsdon on May 18, 2020 | prev | next [–]           
           
 For more queries, you can add modifiers to a Google Search to get the results you want          
           
 Seeing folks mention the NOT operator (-). It's quite powerful! For example, you can do:          
           
 intext:"Powered by intercom" -site:intercom.com will find all the sites that use the Intercom widget          
           
 or ~blog bread baking -inurl:checkout -intext:checkout will find bread blogs (or similar) without commercial intent          
           
 I put together a list of the two dozen or so most useful templates of this, for folks who are interested: https://www.alec.fyi/dorking-how-to-find-anything-on-the-int...         
           
           
 dhbradshaw on May 16, 2020 | prev | next [–]           
           
 I've wondered too about something similar to that. Basically, I'd like sessions for searching.          
           
 Each session would have an updatable list of sites that are favored, whitelisted or blacklisted for a particular class of search.         
           
           
 maayank on May 16, 2020 | prev | next [–]           
           
 I'm intrigued by actual use-cases for it except exploring, i.e. where it would give better result for a query than the common search engines.         
           
 Anyone reading this, please post if you find any           
           
           
 crocodiletears on May 16, 2020 | parent | next [–]           
           
 Big ones:           
           
 1. Looking for niche domain or institutional/social knowledge produced by experts or insiders for an informed audience that isn't necessarily available in a scientific journal.         
           
 Especially with respect to the social sciences and literary analysis, there's a wealth of intelligent commentators that don't surface well on Google without very specific search terms, and the willful subtraction of domains like quora, medium, and tumblr.       
           
 They're usually contained on poorly maintained WordPress sites that the author has long-since forgotten about, or as invalid, handcoded html docs hidden in the personal subdomains of university professors and students.        
           
 2. Finding online communities that aren't a part of Reddit or a similarly prominent platform          
           
           
 chasd00 on May 16, 2020 | prev | next [–]           
           
 in the same vein, it would be awesome to search for a product to buy with the results being ecomm websites owned by people in my area. A way to "shop local" online.         
           
           
 technotarek on May 16, 2020 | parent | next [–]           
           
 ATTIC: A visual search and discovery engine to help you find the latest products from small, unique businesses near you.         
           
 https://attic.city/           
           
 Currently for three product tiers (furniture, home decor, and fashion/clothing) in 14 major US markets, where stores within ~100 miles or a ~2 hour drive are considered as part of the market.        
           
 Disclaimer: I'm one of the founders.           
           
           
 derision on May 16, 2020 | root | parent | next [–]           
           
 How do you curate the stores?           
           
           
 technotarek on May 16, 2020 | root | parent | next [–]           
           
 Aside from the constraints we apply to market/geography and product type? If that's what you mean, then technically it's a matter of whether the store's ecom platform is compatible with our indexer, which supports ~20 different platforms (and hundreds of variations). Otherwise, we do some light curating for product quality to include, but not limited to, the accuracy of meta data (titling, description) and image quality.     
           
           
 fomine3 on May 18, 2020 | parent | prev | next [–]           
           
 I'm curious how English speakers (especially not in US?) finding national shopping websites because I live in Japan so never find foreign website written in Japanese         
           
           
 alibaba_x on May 16, 2020 | parent | prev | next [–]           
           
 Shopify released a mobile app recently for this exact purpose.          
           
           
 technotarek on May 16, 2020 | root | parent | next [–]           
           
 ...but only for stores using Shopify as their ecom platform. That misses A LOT of what's out there, probably 75% or more.         
           
           
 amelius on May 16, 2020 | prev | next [–]           
           
 If only Google allowed us to omit websites from search results.          
           
 Google says they need our information to "improve our experience", but we can't tell them what to omit ...          
           
           
 rsoto on May 16, 2020 | parent | next [–]           
           
 They used to allow that, it was very useful. But as with almost everything Google does, they killed it.          
           
           
 fedede on May 17, 2020 | prev | next [–]           
           
 Hey! I actually liked this idea and I'm considering starting a learning project on it. I've seen a lot of interest and ideas in the comments, and decided to create a very short Google form to start gathering all the interested people so we can organise something interesing. Is anybody in? :)       
           
 https://forms.gle/5KuTYVdYaMzRD2n78           
           
           
 jsgo on May 16, 2020 | prev | next [–]           
           
 I don't know that I'd want a search engine to specifically exclude or limit the results of specific sites of their choosing (even if top 500 as the example is fairly unbiased), but I think I'd really like the ability to say "move these specific domains a few pages back. Don't eliminate them outright, but I have felt dumber having read them previously."      
           
           
 pengstrom on May 16, 2020 | prev | next [–]           
           
 What I want is to filter out commercial results. When I'm searching for a product I don't want shills, I want real opinions.         
           
           
 third_I on May 16, 2020 | parent | next [–]           
           
 And then came undisclosed sponsorship and that difference blurred more than ever...          
           
           
 21xhipster on May 16, 2020 | prev | next [–]           
           
 https://cyber.page           
           
 Its kinda new so it excludes kinda everything :-) But you can make it work better :-)          
           
 https://ipfs.io/ipfs/QmQ1Vong13MDNxixDyUdjniqqEj8sjuNEBYMyhQ...          
           
           
 Aeolun on May 16, 2020 | parent | next [–]           
           
 Can you explain to me what this it is (or is meant to be)? It doesn’t appear to have a search field, though the lightning effects are impressive.         
           
           
 freefriedrice on May 16, 2020 | prev | next [–]           
           
 Why exclude the biggest websites?           
           
 The problem I see on DDG & Google is having to scroll 5-10 pages of utter SEO nonsense.          
           
 "Do you have a question about ____? Many ask about _______. ____ is a common question, here the are we some answer. [sic]".         
           
 Just utter garbage pages.           
           
 It used to be just with recipes or medical questions, but now it feels like most everything that is a general query.          
           
           
 dredmorbius on May 16, 2020 | parent | next [–]           
           
 https://news.ycombinator.com/item?id=22792243           
           
           
 piusp on May 16, 2020 | prev | next [–]           
           
 I have used copernik in the past this was a collection of search engines, listing more than 140 search engines. It combined the search results and sorted them by the key word % matched. It also had a lot of tools inbuilt for validating he links, coping the selection/ sorting and sharing the results. Simply amazing results.      
           
           
 wyck on May 16, 2020 | prev | next [–]           
           
 Google search is so sad these days, all results are media conglomerates, it's completely counter to the core reason why the internet existed. I really hope by catering to these mega corps that they are completely undermining their brand and someone else comes along and pulls the rug out from underneath them.       
           
 If anyone noticed during the first couple days of covid, google search was free from large media results, the algorithm reverted back to how it was years ago and it was such a breath of fresh air. Of course they fixed the algo immediately, it went back to only showing curated media results..there was an anon google employee who posted why this occurred.      
           
           
 fermienrico on May 16, 2020 | parent | next [–]           
           
 I think we are gonna see aversion from going public. Companies like Stripe and SpaceX are gonna stay private for a long time.         
           
 When SEC laws, shareholder interest, quarterly performance and stock volatility comes into play, corporations become this mindless soulless monster that will devour everything in its way and fuck consumers in every which way.        
           
 Democratization of funds from central authority to public creates disincentives and the shareholders don’t give a shit about many auxiliary things such as environmental concerns. Bottom line always matters.        
           
 It’s not just google but any public corporation. Can you imagine SpaceX being able to operate with the same passion with shareholder interests?         
           
           
 texasbigdata on May 16, 2020 | root | parent | next [–]          
           
 It's not, I would argue, those things.           
           
 It is, potentially, the compensation plans. If you go to the proxy document and look at how comp plans are set, they usually hire a consultant, and "best practices" drivers are cash + big bonus based on typically some TSR (total shareholder return metric).       
           
 So for google, "don't be evil" is what's written down, but for the top execs "sell ads" is what gets they paid out before they retire. And those senior level "lifers" are what 40 now?        
           
 Don't really have proof to support these claims though.          
           
           
 Shared404 on May 17, 2020 | root | parent | next [–]           
           
 Not even sure "don't be evil" is still written down. [1]          
           
 [1] https://gizmodo.com/google-removes-nearly-all-mentions-of-do...          
           
           
 koheripbal on May 16, 2020 | parent | prev | next [–]           
           
 I find the only way to get useful results from google is to use the "site:xyz.com" or "related:" options.          
           
           
 om42 on May 16, 2020 | parent | prev | next [–]           
           
 Do you have a link to that thread where an anon google employee posted on why it occurred? Interested in what happened since I didn't realize it did.         
           
           
 wyck on May 18, 2020 | root | parent | next [–]           
           
 The reason given: Legit covid information was not getting through during the fist few days of the crisis, basically made an emergency move to disable the whitelist/blacklist functionality of the algo which reverted to how it worked years ago.        
           
           
 chrischen on May 16, 2020 | parent | prev | next [–]           
           
 It’s a hard problem because the lower down the pyramid you go the more content spam there is.          
           
           
 kian on May 16, 2020 | parent | prev | next [–]           
           
 Do you have a source for the claim in the last paragraph? Would be very interested to read it.          
           
           
 pkamb on May 16, 2020 | prev | next [–]           
           
 I'd love a search engine that mainly searched Stack Exchange, (old.)Reddit, and some subset of blogs or single-author websites.         
           
 Especially removing Quora, Pinterest, and aggregation/reposting/SEO/affiliate blogs.          
           
 And all "product" images with a white background. Only show real photographs.          
           
           
 dredmorbius on May 16, 2020 | parent | next [–]           
           
 Setting specific site restrictions (only one applies at a time, e.g., "site:example.com"), or utilising DDG bang search, gets close.         
           
           
 Cyclone_ on May 16, 2020 | prev | next [–]           
           
 Seems like a browser plugin might be a quick and dirty way of just filtering results to achieve the goal.          
           
           
 social_quotient on May 16, 2020 | prev | next [–]           
           
 A mainstream search engines kinda like a big marker equity ETF or index? There are a ton of benefits but as a negative they make price discovery difficult and give monetary allocation to companies that probably shouldn’t have it.        
           
 Just a thought experiment, curious what others think.           
           
           
 wmnwmn on May 16, 2020 | prev | next [–]           
           
 Maybe what we need is a return to the very beginning, namely human curated web catalogs, aka Yahoo          
           
           
 dluan on May 17, 2020 | prev | next [–]           
           
 I mainly use google as a reddit search engine these days. "tiki cocktail pineapple juice reddit" gives me way more than google algorithm, and plus it's kind of like human powered SEO where genuinely useful links will likely have some discussion.        
           
           
 rdtwo on May 17, 2020 | prev | next [–]           
           
 So I figured I’d try a few of these with “Seattle vegetable garden blog” as the keyword. Either there aren’t a lot of blogs on the topic or most search engines miss them because results are sparse and they really shouldn’t be.        
           
           
 ErikAugust on May 16, 2020 | prev | next [–]           
           
 A curated, searchable web directory might be a concept that could come to be these days. It would share some of its DNA with the old school web directory but also share some with a search engine.        
           
           
 tokyokawasemi on May 18, 2020 | prev | next [–]           
           
 I sometimes use "inurl:wordpress" when searching for travel info. This ensures more first-person blog accounts, rather than all the tripadvisor junk that's at the top.         
           
           
 known on May 16, 2020 | prev | next [–]           
           
 https://twitter.com/search?q=twitter&src=typed_query           
           
           
 moreWeed on May 16, 2020 | prev | next [–]           
           
 Man you read my mind, just starting thinking about this. From a search censorship perspective, the BBS's we were building in 93 would be better than what we have now.         
           
           
 Nevada-Smith on May 16, 2020 | prev | next [–]           
           
 Depending on what you're looking for, try Google Scholar [1]          
           
 [1] https://scholar.google.com/           
           
           
 blondin on May 16, 2020 | prev | next [–]           
           
 omg yes please.           
           
 can google allow us to exclude certain sites? i was surprised to see w3school showing up above official documentations for pandas and numpy. this is simply ridiculous!!         
           
           
 jotm on May 16, 2020 | parent | next [–]           
           
 the "-" operator still works. E.g. "weird stuff I found interesting -reddit.com -youtube.com -wired.com -w3schools.com"         
           
           
 badrabbit on May 16, 2020 | prev | next [–]           
           
 It wouldn't be hard to remove such results using a browser extension,but you will be scrolling a lot. Maybe duckduckgo should support it,feature request?         
           
           
 saadalem on May 16, 2020 | prev | next [–]           
           
 Ok here is an additional idea for fun :           
           
 A search engine that shows only urls that are not indexed b google / another one that gives you the websites with lower pagerank         
           
           
 bmwracer on May 16, 2020 | parent | next [–]           
           
 Not sure it would yield any useful results, but have been thinking about a no-index search engine for a while to help find obscure or otherwise hidden information. If one exists or you build one let me know.        
           
           
 webspinner on May 19, 2020 | parent | prev | next [–]           
           
 Showing all the lower ranked sites would be pretty cool. A lot of blogs would probably end up there, especially if they don't care about pagerank like me.         
           
           
 jungletime on May 16, 2020 | prev | next [–]           
           
 Is there an option to filter out news articles?           
           
 "If you don’t read the newspaper you are uninformed; if you do read the newspaper you are misinformed." Mark Twain          
           
           
 lihaciudaniel on May 16, 2020 | parent | next [–]           
           
 That's why I watch TV instead.           
           
           
 dangoljames on May 16, 2020 | prev | next [–]           
           
 Is there a search engine that actually combines selective search logic with reductive logic, so that can be used to actually search topically?         
           
           
 corndoge on May 16, 2020 | prev | next [–]           
           
 Similarly, I have always wanted a YouTube search filter for "least views", since that content is invariably way higher quality         
           
           
 thoughtstheseus on May 16, 2020 | prev | next [–]           
           
 I think one of the underlying problems in search is that most search engines are more like recommendation systems.          
           
           
 runawaybottle on May 16, 2020 | prev | next [–]           
           
 Filter google against Alexa rankings?           
           
           
 egberts1 on May 17, 2020 | prev | next [–]           
           
 What we all really need is a long-tail Bloom filter search engine on the search engines themselves.          
           
           
 coronadisaster on May 17, 2020 | prev | next [–]           
           
 if google would switch to the google's engine code that was used right before they modified the "+" operator for google-plus, it would be a lot better.... ie: bring back the + operator please (the quotes dont work the same way)        
           
           
 citizenpaul on May 18, 2020 | prev | next [–]           
           
 RSS used to be this. Google has done its best to kill it.          
           
           
 aiisjustanif on May 17, 2020 | prev | next [–]           
           
 RIP StumbleUpon. The randomized search engine I want back.          
           
           
 starfallg on May 16, 2020 | prev | next [–]           
           
 The elephant in the room is Baidu.           
           
           
 dpau on May 16, 2020 | parent | next [–]           
           
 Ask HN: Is there a search engine which includes only what the Chinese government wants me to see?          
           
           
 starfallg on May 16, 2020 | root | parent | next [–]           
           
 That was tongue-in-cheek, but I guess some people are legitimately offended enough to downvote.          
           
           
 jotm on May 16, 2020 | root | parent | prev | next [–]           
           
 What are they blocking anyway? Anything important?           
           
 There's so many Chinese forums for hardware/firmware hacking/mods, a shame translators are still very bad...          
           
           
 cameronbrown on May 16, 2020 | root | parent | next [–]          
           
 Definitely. What if I want to research information on a certain 1989 student protest to write a paper about censorship?         
           
           
 jotm on May 16, 2020 | root | parent | next [–]           
           
 Look, I know it's important history and all, but looking at it realistically, it's a tiny part of the information the average person needs. Them removing Reddit, EdX, Sci-hub or the Hong Kong Companies Registry from search results would have a much bigger impact.       
           
           
 starfallg on May 22, 2020 | root | parent | next [–]           
           
 You very conveniently left out Wikipedia and all of the other sites that are censored.          
           
           
 wojtczyk on May 16, 2020 | prev | next [–]           
           
 That’s what hacker news is for ;)           
           
           
 Upvoter33 on May 16, 2020 | prev | next [–]           
           
 google: search terms -site:cnn.com -site:wikipedia.org -site:...          
           
           
 aiisjustanif on May 17, 2020 | prev | next [–]           
           
 RIP StumbleUpon.           
           
           
 martin-adams on May 17, 2020 | prev | next [–]           
           
 I like this question. I’ve often wanted a search engine which gives you the choice to find sites that don’t contain a paywall, tracking or advertising.         
           
           
 graycat on May 16, 2020 | prev | next [–]           
           
 For           
           
 > Ask HN: Is there a search engine which excludes the world's biggest websites?          
           
 > Discovering unknown paths of the web seems almost impossible with google et al..          
           
 > Are there any earch engines which exclude or at least penalize results from, say, top 500 websites?          
           
 Let's back up a little and then try for an answer:           
           
 Some points:           
           
 (1) For some qualitative exclamation, there is a LOT of content on the Internet.          
           
 (2) There are in principle and no doubt so far significantly in practice a LOT of searches people want to do. The search in the OP is an example.         
           
 (3) Much like in an old library card catalog subject index, the most popular search engines are based heavily on key words and then whatever else, e.g., page rank, date, etc.         
           
 So: (1) -- (3) represent some challenges so far not very well met: In particular, we can't expect that the key words, etc. of (3) will do very well on all or nearly all the searches in (2) for much of the content in (1).        
           
 And the search in the OP is an example of a challenge so far not well met.          
           
 Moreover, the search in the OP is no doubt just one of many searches with challenges so far not well met.          
           
 Long ago, Dad had a friend who worked at Battelle, and IIRC they did a review of information retrieval that concluded that keyword search covers only a fraction, maybe ballpark only 1/3rd, of the need for effective searching. And the search in the OP is an example of what is not covered because the library card catalog did not index size of the book or Web site! :-)!      
           
 Seeing this situation, my rough, ballpark estimate has been that the currently popular Internet search engines do well on only about 1/3rd of the content on the Internet, searches people want to do, and results they want to find.        
           
 So, I decided to see what could be done for the other 2/3rds.          
           
 I started with some not very well known or appreciated advanced pure math; it looks like useless, generalized abstract nonsense, but if calm down, stare at it, think about it, ..., can see a path for a solution. Although I never thought about the search in the OP until now, in principle the solution should work also for that search. Or, the math is a bit abstract and general which can translate in practice to doing well on something as varied as the 2/3rds.    
           
 Then for the computing, I did some original applied math research.          
           
 Using TeX, I wrote it all up with theorems and proofs.           
           
 So, the project is to be a Web site. While in my career I've been programming for decades, this was my first Web site. I selected Windows and .NET, and typed in 100,000 lines of text with 24,000 statements in Visual Basic .NET (apparently equivalent in semantics to C# but with syntactic sugar I prefer).       
           
 The software appears to run as intended and well enough for significant production.          
           
 I was slowed down by one interruption after another, none related to the work.          
           
 But, roughly, ballpark, the Web site should be good, or by a lot the best so far, for the 2/3rds and in particular for the search in the OP.         
           
 So, for           
           
 > Ask HN: Is there a search engine which excludes the world's biggest websites?          
           
 there's one coded and running and on the way to going live!          
           
 I intend to announce an alpha test here at HN.           
           
           
 defen on May 17, 2020 | parent | next [–]           
           
 Can you talk at a high level about the problems of keyword search, or is that part of your secret sauce? Off the top of my head I can think of two, which are intent and encoding .         
           
 Before you can even do a keyword search, you obviously need an intent to do so. But that means keyword search is pretty useless when you don't know what you don't know.         
           
 Encoding that intent...maybe doesn't matter for common searches, but everyone has heard of the concept of "Google-Fu". English text is a pretty lossy medium compared to the thoughts in people's heads...Shannon calculated 2.62 bits per English letter, so the space of possibly-relevant sites for almost any keyword is absolutely enormous (e.g. there are about 330,000 7-letter english keyword searchs...distributed across how many trillions of pages, not even counting "deep web" dynamically generated ones?). So we punt on that and use the concept of relevance for sorting results, and in practice no one looks beyond the first 10. I don't know what an alternate encoding might look like though 
           
           
 graycat on May 17, 2020 | root | parent | next [–]           
           
 Good questions.           
           
 For           
           
 > Before you can even do a keyword search, you obviously need an intent to do so. But that means keyword search is pretty useless when you don't know what you don't know.         
           
 Right: The way at times in the past I have put something like that is to say that, ballpark, to oversimplify some, keyword search requires the user to know what content they want, know that it exists, and have keywords/phrases that accurately characterize that content. For some searches, e.g., the famous movie line      
           
 "I don't have to show you no stinking badges",           
           
 https://www.youtube.com/watch?v=VqomZQMZQCQ           
           
 that is fine; otherwise it asks too much of the user.           
           
 For "encoding", my work does not use keywords or any natural language for anything.          
           
 My work does get some new data for each search for each user. But privacy is relatively good because for the results I use only what the user gives for that search; in particular, two users giving the same inputs at essentially the same time will get the same results. Thus, search results are independent of the user's IP address or browser agent string. Moreover, the site makes no use of cookies.     
           
 The role of the advanced pure math is to say that the data I get and the processing I do with that data and what is in the database should yield good results for the 2/3rds. The role of my original applied math is to make the computations many times faster -- they would be too slow otherwise.       
           
 When keywords work well, and they work well enough to be revolutionary for the world, my work is, except for some small fraction of cases, not better. So, there is ballpark the 1/3rd where keywords work well. Then there is the ballpark, guesstimate, 2/3rds I'm going for.       
           
 My work is not as easy for the users as picking a great, very accurate, result from the top dozen presented by a keyword search engine, e.g., the movie line example, but is much easier to use than flipping through 50 pages of search results and is intended usually to give good results unreasonable to get from a keyword search, without "characterizing" keywords, that yields, say, millions of search results and would require a user to flip through dozens of pages of search results.    
           
 Ads are off on the right side and not embedded in the search results. The SEO (search engine optimization) people will have a tough time influencing the search results!         
           
 We will see how well users like it. If people like it, then it will be good to make progress on the huge, usually neglected, content of the 2/3rds.         
           
           
 Gollapalli on May 17, 2020 | parent | prev | next [–]           
           
 Looking forward to it. You got a name?           
           
           
 graycat on May 17, 2020 | root | parent | next [–]           
           
 I have a list of 100+ candidates but (i) I have not selected one yet, (ii) don't really need one before I register it, and (iii) don't want to start paying for it before I need it.         
           
 But thanks for your interest.           
           
           
 notaphilosopher on May 17, 2020 | prev | next [–]           
           
 I'd like:           
           
 - health search that excludes sellers, wellness and snake-oil websites          
           
 - news search that excludes conspiracy theories, magical thinking, political operatives, and paid bloggers          
           
 - image search by similarity, similarity to an uploaded picture/s, words, or description          
           
 - media and warez search engine that excludes link-spam and malware sites          
           
 - complex queries search because none of them do it well          
           
 - anonymity           
           
 - shopping search that kicks out disreputable sellers and phony store-fronts          
           
 - mapping like OSM but fast, practical with an app, and detail-accessible          
           
 - monetize using affiliate links that don't affect ranking          
           
 - semi-curated results (domain reputation-ranked voting)          
           
 - related pages           
           
 - inbound/outbound links search           
           
 - archive.org integration &| history page caching           
           
 - documented query syntax           
           
 - query within results           
           
 - quick query history results navigation           
           
 - keyword alerts           
           
 - keyboard shortcuts that always work           
           
           
 burmer on May 16, 2020 | prev [–]           
           
 DuckDuckGo /s           
           
           
           
           
           
           
           
           
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact          
           
Search:           

Discovering unknown paths of the web seems almost impossible with google et al.. Are there any earch engines which exclude or at least penalize results from, say, top 500 websites?
Ask HN: Is there a search engine which excludes the world's biggest websites?
https://news.ycombinator.com/item?id=23202850
https://news.ycombinator.com/item?id=28550764
A search engine that favors text-heavy sites and punishes modern web design
2021-09-16
None
<div><div><td><table>
 <tbody><tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552193&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Wow, that's awesome. Great work!<p>For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources.</p><p>When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totally useless.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553521&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Good comparison. Reminds me of an analogy I like to make of today's web, which is it feels like browsing through a magazine store — full of top 10s, shallow wow-factoids, and baity material. I genuinely believe terrible results like this are making society dumber.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555154&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>The context matters. I'd happily read "Top 10" lists on a website if the site itself was dedicated to that one thing. "Top 10 Prog Rock albums", while a lazy, SEO-bait title, would at least be credible if it were on a music-oriented website.<p>But no, these stories all come from cookie-cutter "new media" blog sites, written by an anonymous content writer who's repackaged Wikipedia/Discogs info into Buzzfeed-style copy writing designed to get people to "share to Twitter/FB". No passion, no expertise. Just eyeballs at any cost.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555585&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>This got me thinking that maybe one of the other big reasons for this is that the algorithms prioritize newer pages over older pages. This produces the problem where instead of covering a topic and refining it over time, the incentive is to repackage it over and over again.<p>It reminds me of an annoyance I have with the Kindle store. If I wanted to find a book on, let's say, Psychology, there is no option to find all-time respected books of the past centenary. Amazon's algorithms constantly push to recommend the latest hot book of the year. But I don't want that. A year is not enough time to have society determine if the material withstands time. I want something that has stood the test of time and is recommended by reputable institutions.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28561522&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>This is just a guess, but I believe that they use machine learning and rank it by the clicks. I took some coursera courses and Andrew Ng sort of suggested that as their strategy.<p>The problem is that clickbait and low effort articles could be good enough to get the click, but low effort enough to drag society into the gutter. As time passes, the system is gamified more and more where the least effort for the most clicks is optimized.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28574714&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>But they have. or could have. At least Google (and to a smaller extend Microsoft), if you are using Chrome/Bing have exactly that signal. If you stay on the site and scroll (taking time, reading, not skimming) all this could be a signal to evaluate if the search result met your needs.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28575364&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I've heard google would guess with bounce rate. Or another way, if the user clicks on LinkedIn website A, after a few moments keeps trying other linksw/related search. It would mean it was not valuable.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555899&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; is that the algorithms prioritize newer pages over older pages.<p>They do? That would explain a lot - but ironically, I can't find a good source on this. Do you have one at hand?
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28556079&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>It is pretty obvious if you search for any old topic that is also covered incessantly by the news. "royal family" is a good example. There's no way those news stories published an hour ago are listed first due to a high PageRank score (which necessarily depends on time to accumulate inbound links).</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28557285&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>It depends on the content. The flip side is looking up a programming-related question and getting results from 2012.<p>I think they take different things into account based on the thing being searched.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28560054&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Even your example would depend upon the context. There are many cases where a programming question in 2021 is identical to one from 2012, along with the answer. In those instances, would you rather a shallow answer from 2021 or an indepth answer from 2012? This is not meant to imply that older answers offer greater depth, yet a heavy bias towards recent material can produce that outcome in some circumstances.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28562110&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>If you're using tools/languages that change rapidly (like Kotlin, in my case), syntax from a few years ago will often be outdated.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563137&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Yes, yet there are programming questions that go beyond "how do I do X in language Y" or "how do I do X with library Y". The language and library specific questions are the ones where I would be less inclined to want additional depth anyhow, well, provided they aren't dependent upon some language or library specific implementation detail.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28573681&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>There are of course a variety of factors, including the popularity of the site the page is published on. The signals related to the site are often as important as the content on the page itself. Even different parts of the same site can lend varying weight to something published in that section.<p>Engagement, as measured in clicks and time spent on page, plays a big part.</p><p>But you're right, to a degree, as frequently updated pages can rank higher in many areas. A newly published page has been recently updated.</p><p>A lot depends on the (algorithmically perceived) topic too. Where news is concerned, you're completely right, algos are always going to favor newer content unless your search terms specify otherwise.</p><p>PageRank, in it's original form, is long dead. Inbound link related signals are much more complex and contextual now, and other types of signals get more weight.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28558559&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Your Google search results show the date on articles do they not? If people are more likely to click on "Celebrity Net Worth (2021)" than "Celebrity Net Worth (2012)", then the algo will update to favour those results, because people are clicking on them.<p>The only definitive source on this would be the gatekeeper itself. But Google never says anything explicitly, because they don't want people gaming search rankings. Even though it happens anyway.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28559410&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>The new evergreen is refreshed sludge for bottom dollar. College kids stealing Reddit comments or moving around paragraphs from old articles. Or linking to linked blogs that link elsewhere.<p>It's all stamped with Google Ads, of course, and then Google ranks these pages high enough to rake in eyeballs and ad dollars.</p><p>Also there's the fact that each year, the average webpage picks up two more video elements / ad players, one or two more ad overlays, a cookie banner, and half a dozen banner/interstitials. It's 3-5% content spread thinly over an ad engine.</p><p>The Google web is about squeezing ads down your throat.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28559528&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Really makes you wonder: you play whack a mole and tackle the symptoms with initiatives like this search engine. But the root of that problem and many many others is the same: advertising. Why don't we try to tackle that?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28570280&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; This got me thinking that maybe one of the other big reasons for this is that the algorithms prioritize newer pages over older pages.<p>Actually that's not always the case. We publish a lot of blog content and it's really hard to publish new content that replaces old articles. We still see articles from 2017 coming up as more popular than newer, better treatments of the same subject. If somebody knows the SEO magic to get around this I'm all ears.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28560155&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Its the "healthy web" Mozilla^1 and Google keep telling their blog audiences about. :)<p>1 Accept quid pro quo to send all queries to Google by default</p><p>If what these companies were telling their readers was true, i.e., that advertising is "essential" for the web to survive, then how are the sites returned by this search engine for text-heavy websites (that are not discoverable through Google, the default search engine for Chrome, Firefox, etc.) able to remain online. Advertising is essential for the "tech" company middleman business to survive.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563307&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I'm not sure I agree with your example. It seems to me it is the exact same as a "Top ten drinks to drink on a rainy day" list. There's simply too many good albums and opinions differ, so a top ten would -just like the drinks- end up being a list of the most popular ones with maybe one the author picks to stir some controversy or discussion. In my opinion the world would be a smarter place if Google ranked all such sites low. Then we might at least get fluff like "Top ten prog rock albums if you love X, hate Y and listen to Z when no one is around" instead.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28565207&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Google won't rank them low because they actually do serve an important purpose. They're there for people who don't really know what they want specifically, they're looking for an overview. A top 10 gives a digestible overview on some topic, which helps the searcher narrow down what they really want.<p>A "Top 10 albums of all time" post is actually better off going through 10 genres of popular music from the past 50 years and picking the top album (plus mentioning some other top albums in the genre) for each one.</p><p>That gives the user the overview they're probably looking for, whether those are the top 10 albums of all time or not. It's a case of what the user searched for vs what they actually really want.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28578715&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>So did Tim Berners-Lee. He was vehemently opposed to people shoehorning images into the WWW, because he didn't want it to turn into the equivalent of magazines. Which, I believe, he shared the opinion of them making society dumber.<p>Appropriately enough, I couldn't find a good quote to verify that since Google is only giving me newspapers and magazines talking about Sir Tim in the context of current events. I do believe it's in his book "Weaving the Web" though.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554726&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>It's also possible that it's the other way around: a certain "common denominator" + algorithms that chase broad engagement = mediocre results.<p>The real trick would be some kind of engine that can aim just above where the user's at.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28566347&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; I genuinely believe terrible results like this are making society dumber.<p>You have to e causality reversed. Google results reflect the fact that society is dumb.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28557335&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>what I really want is a true AI to search through all that and figure out the useful truth. I don't know how to do this (and of course whoever writes the AI needs to be unbiased...)</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28560756&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt;whoever writes the AI needs to be unbiased...)<p>I'm not sure the idea of a sentient being not having a bias is meaningful. Reality, once you get past the trivial bits, is subjective.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28566439&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Isn't there a fundamental ML postulate that learning without bias is impossible?<p>Maybe not the same kind of bias we think of in terms of politics and such, but I wonder if there's a connection.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28564958&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I didn't say the AI should be unbiased, just whoever writes it.<p>I want an AI that is biased to the truth when there is an objective one, and my tastes otherwise. (that is when asked to find a good book it should give me fantasy even though romance is the most popular genre and so will have better reviews)
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28560098&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I think that is the goal, it's just what we currently have is an AI that's like a naive child who is easily tricked and distracted by clickbait.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563361&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt;an AI that's &lt;snip&gt; easily tricked and distracted by clickbait.<p>So, AIs are actually on par with most adults now? (Sorry)
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554478&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Cool, it appears that the trend towards JS may be causing self-selection -- if a page has a high amount of JS, it is highly unlikely to contain anything of value.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28556999&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>True. Unfortunately many large corporate websites through which you pay bills, order tickets, etc. are becoming infested with JS widgets and bulky, slow interfaces. These are hard to avoid.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28562359&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Conversely no software to install. Browser as a platform. Don’t have to boot to Windows to pay your bills with activex for example</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28568605&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>The mostly JS-less web was fine, fast, and reliable 20 years ago and I never had ActiveX.<p>I hear stories about Flash and ActiveX but I literally never needed these to shop or pay bills online. Payments also didn't require scripts from a dozen domains and four redirects..
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28566305&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>The platform isn't the problem. The problem is with the amount of code that does something other than letting you "pay bills, order tickets, etc.".</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563391&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Huh. A weighted algorithm, somewhere between Google and the one linked, where you could subtract from sites by amount of JavaScript might be interesting.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554855&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>If one could create an metric of ad to content ratio from the js used, I would guess that would be a nice differentiator too.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28561098&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Browsers should be cherry picking the most compelling things that people accomplish with complex code and supporting them as a native feature.
Maybe the Browser Wars aren’t keeping up anymore.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552494&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>However, when searching for "haskell type inference algorithm" I get completely useless results.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552843&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>That query is too long apparently. But if you shorten to "haskell type inference", I think it delivers on its promise:<p>&gt; If you are looking for fact, this is almost certainly the wrong tool. If you are looking for serendipity, you're on the right track. When was the last time you just stumbled onto something interesting, by the way?
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553397&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>The search engine doesn't do any type of re-ordering or synonym stuff, it only tires to construct different N-grams from the search query.<p>So if you for example compare "SDL tutorial" with "SDL tutorials". On google you'd get the same stuff, this search engine, for better or worse doesn't.</p><p>This is a design decision, for now anyway, mostly because I'm incredibly annoyed when algorithms are second-guessing me. On the other hand, it does mean you sometimes have to try different searches to get relevant results.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563859&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I’m not against a stemmer, actually, just against the aggressive concordances (?) that Google now employs, like when it shows me X in Banach spaces (the classical, textbook case) when I’m specifically searching for X in Fréchet spaces (the generalization I want to find 
but am not sure exists); of <i>course</i> Banach spaces and Fréchet spaces are almost exclusively encountered in the same context, but it doesn’t mean that one is a popular typo for the other! (The relative rarity of both of these in the corpus probably doesn’t help. The farcical case is BRST, or Becchi-Rouet-Stora-Tyutin, in physics, as it is literally a single key away from “best” and thus almost impossible to search for.)<p>On the other hand, Google’s unawareness of (extensive and ubiquitous) Russian noun morphology is essentially what allowed Yandex to exist: both 2011 Yandex and 2021 Google are <i>much</i> more helpful for Russian than 2011 Google. I suspect (but have not checked) that the engine under discussion is utterly unusable for it. English (along with other Germanic and Romance languages to a lesser extent) is quite unusual in being meaningfully searchable without any understanding of morphology, globally speaking.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28564441&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I thought you could fix that by enclosing "BRST" in quotes, but apparently not. DuckDuckGo (which uses Google) returns a couple of results that do contain "BRST" in a medical context, but most results don't contain this string at all. What's going on?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28564720&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I’m not certain what DDG actually uses (wasn’t it Bing?), but in my experience from the last couple of months it ignores quotes substantially <i>more</i> eagerly than Google does. For this particular term, a little bit of domain knowledge helps: even without quotes, <i>brst becchi</i>, <i>brst formalism</i>, <i>brst quantization</i> or perhaps <i>bv brst</i> will get you reasonable results. (I could swear Google corrected <i>brst quantization</i> to <i>best quantization</i> a year ago, but apparently not anymore.) Searching for stuff in the context of BRST is still somewhat unpleasant, though.<p>I... don’t think anything particularly surprising is happening here, except for quotes being apparently ignored? I’ve had it explained to me that a rare word is essentially indistinguishable from a popular misspelling by NLP techniques as they currently exist, except by feeding the machine a massive dictionary (and perhaps not even then). BRST is a thing that you essentially can’t even define satisfactorily without at the very least four years of university-level physics (going by the conventional broad approach—the most direct possible road can of course be shorter if not necessarily more illuminating). “Best” is a very popular word both generally and in searches, and the R key is next to E on a Latin keyboard. If you are a perfect probabilistic reasoner with only these facts for context (and especially if you ignore case), I can very well believe that your best possible course of action is to assume a typo.</p><p>How to permit overriding that decision (and indeed how to recognize you’ve actually made one worth worrying about without massive human input—<i>e.g.</i> Russian adjectives can have more than 20 distinct forms, can be made up on the spot by following productive word-formation processes, and you don’t want to learn all of the world’s languages!) is simply a very difficult problem for what is probably a marginal benefit in the grand scheme of things.</p><p>I just dislike hitting these margins so much.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28565278&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>It would not be a difficult problem if they allowed the " " operator to work as they claim it does, or revive the + operator.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28565737&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>In English, maybe; in Russian, I frequently find myself reaching for the nonexistent “morphology but not synonyms” operator (as the same noun phrase can take a different form depending on whether it is the subject or the object of a verb, or even on which verb it is the object of); even German should have the same problem AFAIU, if a bit milder. I don’t dare think about how speakers of agglunative languages (Finnish, Turkish, Malayalam) suffer.<p>(DDG docs do say it supports +... and even +"...", but I can’t seem to get them to do what I want.)
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28566302&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Ah, OK. I don’t know anything about Russian. This is a hard problem. I think the solution is something like what you suggest: more operators allowing different transformations. Even in English, I would like a "you may pluralize but nothing else" operator.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28572465&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Well it’s not that alien, it (along with the other Eastern Slavic languages, Ukrainian and Belarusian) is mostly a run-of-the-mill European language (unlike Finnish, Estonian or Hungarian) except it didn’t lose the Indo-European noun case system like most but instead developed even more cases. That is, where English or French would differentiate the roles of different arguments of a verb by prepositions or implicitly by position, Russian (like German and Latin) has a special axis of noun forms called “case” which it uses for that (and also prepositions, which now require a certain case as well—a noun form can’t <i>not</i> have a case like it can’t <i>not</i> have a number).<p>There are six of them (nominal [subject], genitive [belonging, part, absence, “of”], dative [indirect object, recipient, “to”], accusative [direct object], instrumental [device, means, “by”], prepositional [what the hell even is this]), so you have (cases) × (numbers) = 6 × 2 = 12 noun forms, and adjectives agree in number and gender with their noun, but (unlike Romance languages) plurals don’t have gender, so you have (cases) × (numbers and genders) = 6 × (3 + 1) = 24 adjective forms.</p><p>None of this would be particularly problematic, except these forms work like French or Spanish verbs: they are synthetic (case, number and gender are all a single fused ending, not orthogonal ones) and highly convoluted with a lot of irregularities. And nouns and adjectives are usually more important for a web search than verbs.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28572597&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Well yeah, English is kind of weird, but Finnish isn’t a Germanic language at all? It’s not even Indo-European, so even Hindi is ostensibly closer to English than Finnish. I understand Standard German (along with Icelandic) is itself a bit atypical in that it hasn’t lost its cases when most other Germanic languages did.<p>Re compounds, I expected they would be more or less easy to deal with by relatively dumb splitting, similar to greedy solutions to the “no spaces” problem of Chinese and Japanese, and your link seems to bear that out. But yeah, cheers to more language-specific stuff in your indexing. /s
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28562487&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Oh this sounds like it could be a really cool idea! This way it could also be subtly teaching users that the engine doesn't do automatic synonyms translation so it's worth experimenting; also kinda like giving the synonyms feature while still keeping user in full control.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552538&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Since it does not use synonyms, it looks like it is unable to answer "how's that thing called"-queries.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553576&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Definitely; We could create a meta search engine that queries them all, in desktop application format.<p>Let's name it after a famous old scientist, and maybe add the year to prove it's modern: Galileo 2021.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555360&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; You can't combine a few different ranked lists and expect to get results better than any of the original ranked lists.<p>I am skeptical of this application of the theorem. Here is my proposal:</p><p>Take the top 10 Google and Bing results. If the top result from Bing is in the top 10 from Google, display Google results. If the top result from Bing is not in the top 10 from Google, place it at the 10th position. You'd have an algorithm that ties with Google, say 98% of the time, beats it say, 1.2% of the time, and loses .8% of the time.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555771&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Right. Arrow's theorem just says it's impossible to do it in <i>all</i> cases. It's still quite possible to get an improvement in a large proportion of cases, as you're proposing.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28556851&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I've had jobs tuning up the relevance of search engines with methods like<p><a href="https://ccc.inaoep.mx/~villasen/bib/AN%20OVERVIEW%20OF%20EVALUATION%20METHODS%20IN%20TREC%20AD%20HOC%20IR%20AND%20TREC%20QA.pdf">https://ccc.inaoep.mx/~villasen/bib/AN%20OVERVIEW%20OF%20EVA...</a></p><p>and the first conclusion is "something that you think will improve relevance probably won't"; the TREC conference went for about five years before making the first real discovery</p><p><a href="https://en.wikipedia.org/wiki/Okapi_BM25">https://en.wikipedia.org/wiki/Okapi_BM25</a></p><p>It's true that Arrow's Theorem doesn't strictly apply, but thinking about it makes it clear that the aggregation problem is ill-defined and tricky. (e.g. note also that a ranking function for full text search might have a range of 0-1 but is not a meaningful number, like a probability estimate that a document is relevant, but it just means that a result with a higher score is likely to be more relevant than one with a lower score.)</p><p>Another way to think about it is that for any given feature architecture (say "bag of words") there is an (unknown) ideal ranking function.</p><p>You might think that a real ranking function is the ideal ranking function plus an error and that averaging several ranking functions would keep the contribution of the ideal ranking function and the errors would average out, but actually the errors are correlated.</p><p>In the case of BM25 for instance, it turns out you have to carefully tune between the biases of "long documents get more hits because they have more words in them" and "short documents rank higher because the document vectors are spiky like the query the vectors". Until BM25 there wasn't a function that could be tuned up properly and just averaging several bad functions doesn't solve the real problem.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28558355&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Arrows theorem simply doesn't apply here. We don't need our personalized search results to satisfy the majority.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28558803&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>But in both cases you face the problem of aggregating preferences of many into one. In one case you are combining personal preferences in the other case aggregating ‘preferences’ expressed by search engines.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28567135&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>But search engines aren't voting to maximize the chances that their preferred candidate shows up on top. The mixed ranker has no requirement to satisfy Arrows integrity constraints. It has to satisfy the end user, which is quite possible in theory.<p>Conditions the mixed ranker doesn't have to satisfy 
"ranking while also meeting a specified set of criteria: unrestricted domain, non-dictatorship, Pareto efficiency, and independence of irrelevant alternatives"
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28569156&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Sure, but the problem that conventional IR ranking functions are not meaningful other than by ordering leads you to the dismal world of political economy where you can't aggregate people's utility functions. (Thus you can't say anything about inequality, only about Pareto efficiency)<p>Hypothetically you could treat these functions as meaningful but when you try you find that they aren't very meaningful.</p><p>For instance IBM Watson aggregated multiple search sources by converting all the relevance scores to "the probability that this result is relevant".</p><p>A conventional search engine will do horribly in that respect, you can fit a logit curve to make a probability estimator and you might get p=0.7 at the most and very rarely get that, in fact, you rarely get p&gt;0.5.</p><p>If you are combining search results from search engines that use similar approaches you know those p's are not independent so you can't take a large numbers of p=0.7's and turn that into a higher p.</p><p>If you are using search engines that use radically different matching strategies (say they return only p=0.99 results with low recall) the Watson approach works, but you need a big team to develop a long tail of matching strategies.</p><p>If you had a good p-estimator for search you could do all sorts of things that normal search engines do poorly, such as "get an email when a p&gt;0.5 document is added to the collection."</p><p>For now alerting features are either absent or useless and most people have no idea why.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28556648&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>That's an invalid application of this theorem. (It doesn't necessarily hold)<p>Suppose there's an unambiguous ranked preference by all people among a set (webpages, ranking). Suppose one search engine ranks correctly the top 5 results and incorrectly the next 5 results, while another ranks incorrectly the top 5 and correctly the next 5.</p><p>What can happen is that some there may be no universally preferred search engine (likely). In practice, as another commenter noted, you can also have most users prefer more a certain combination of results (that's not difficult to imagine, for example by combining top independent results from different engines for example).
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555486&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Yup, but so far no one got it.<p>There was such an app in the early 2000's, before Google went mainstream, and Altavista-like engines were not good: Copernic 2000.</p><p>I guess I'm officially old now.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28559004&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>For years I wanted to try Copernic Summarizer. It seemed like it actually worked. Then software that did summaries disappeared, maybe? And about 5 years ago bots on Reddit were doing summaries of news stories (and then links in comments).<p>This is a pattern I see over and over again, some research group or academics show that something can be done (summaries that make sense and are true summaries, evolutionary algorithm FPGA programming, real time gaze prediction, etc) and there's a few published code repos and a bit of news, then 'poof' - no where to be seen for 15 years or more.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554930&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p><i>a magic pop sound is faintly audible as a new side project is appended to several lists</i> Excellent, thank you!</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552344&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>As long as few people use it, it will be great. Rest assured that the moment it becomes popular, the people who want to game it will appear.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552628&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>This sort of optimization is why simple recipes are typically found at the end of a rambling pointless blog post now.<p>Still, the best way to break SEO is to have actual competition in the search space. As long as SEO remains focused on Google there is an opportunity for these companies to thrive by evading SEO braindamage.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553426&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>That sort of recipe blog hasn't happened just for SEO. It's also a bit of a "two audiences" problem: if you are coming to that food blogger from a search you certainly would prefer the recipe first and then maybe any commentary on it below if the recipe looks good. If you are a regular reader of that food blogger you are probably invested in the stories up top and that parasocial connection and the recipes themselves are sometimes incidental to why you are a regular reader.<p>You see some of that "two readers" divide sometimes even in classic cookbooks, where "celebrity" chefs of the day might spend much of a cookbook on a long rambling memoir. Admittedly such books were generally well indexed and had table of contents to jump right to the recipes or particular recipes, but the concept of "long personal ramble of what these recipes mean to me" is an old one in cookbooks too.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554197&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I see your point, but argue you've misidentified the two audiences.<p>One audience matches your description and is the invested reader. They want <i>that</i> blogger's story telling. they might make the recipe, but they're a dedicated reader.</p><p>The other audience is not the recipe-searcher, but instead Google. Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on. They won't even remember the blog's name. So the site isn't optimized for them. It's optimized for Google.</p><p>"Slow the parasitic recipe-searcher down. They're leeches, here for a freebie. Well they'll pay me in Google Rank time blocks."
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28574825&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on.<p>This is not entirely true, though. If a randomly found recipe turns out particularly good, I'll bookmark the site and try out other dishes. It's a very practical method to find particularly good* recipe collections.</p><p>*) In this case "good" means what you need - not just subjectively "tasty", but e.g. low cost, quick to prepare, low calorie or in line with a particular diet and so on.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28556550&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; If you are a regular reader of that food blogger<p>I think this assumes facts not in evidence. It certainly seems like an overwhelming number of "blogs" are not actual blogs but SEO content farms. There's no regular readers of such things because there's no actual authors, just someone that took a job on Fivver to spew out some SEO garbage. Old content gets reposted almost verbatim because new results better according to Google.</p><p>The only reason these "blogs" exist is to show ads and hopefully get someone's e-mail (and implied consent) for a marke....newsletter.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28557282&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I know at least a few that I commonly see in top search results that I have friends that read them like personalized soap operas where most of the drama revolves around food and family and serving food to family.<p>It's at least half the business models of Food Network shows: aspirational kitchens and the people that live in them and also sometimes here's their recipes. (The other half being competitions, obviously.) I've got friends that could deliver entire doctoral theses on the Bon Appetit Test Kitchen (and its many YouTube shows and blogs) and the huge soap operatic drama of 2020's events where the entire brand milkshake ducked itself; falling into people's hearts as "feel good" entertainment early in 2020/the pandemic and then exploding very dramatically with revelations and betrayals that Fall.</p><p>Which isn't to say that there <i>aren't</i> garbage SEO farms out there in the food blogging space <i>as well</i>, but a lot of the big ones people commonly complain about seeing in google's results do have regular fans/audiences. (ETA: And many of the smaller blogs <i>want</i> to have regular fans/audiences. It's an active influencer/"content creator" space with relatively low barrier to entry that people love. Everyone's family loves food, it's a part of the human condition.)
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554068&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I've basically never been taken to a recipe without a rambling preamble from Google. While food blogs may serve two audiences, a long introduction seems to be a requirement to appear in the top Google search results.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554153&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Personally, I think that has a lot more to do with the fact that Google killed the Recipe Databases. There did used to be a few startups that tried to be Recipe Aggregators with advertising based business models, that would show recipes and then link to source blogs and/or cookbooks, and in the brief period where they existed Google scraped them <i>entirely</i> and showed entire recipes on search results and ate their ad revenue out from under them.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563167&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Such databases would get battered by demands to remove content these days, if not already back then. No one want a database listing their stuff for ad revenue like that because many wouldn't follow the links so see <i>their</i> adverts or be subject to <i>their</i> tracking.<p>A couple of browser add-ons specifically geared around trimming recipe pages down have been taken down due to similar complaints.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555139&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Their core business is advertising and they have always been in a direct conflict-of-interest by competing with content sites for ad revenue buys.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552682&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>That's why I use Saffron [1], it magically converts those sites into a page in my recipe book. I found it when the developer commented here in HN. Also, a lot of cooking website have started to add a link with "jump to recipe" functionality allowing you to skip all the crap.<p>[1] <a href="https://www.mysaffronapp.com/">https://www.mysaffronapp.com/</a>
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552743&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I've noticed this pattern start to pop up elsewhere. I've started to train my skimming skills, skipping a paragraph or two at a time to get past the fluff.<p>Like an article about some current event will undoubtedly begin with "when I was traveling ten years ago...".
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555598&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>It's also because that's a way of trying to copyright protect recipes, which are normally not copyright protected.<p>&gt; “Mere listings of ingredients as in recipes, formulas, compounds, or prescriptions are not subject to copyright protection. However, when a recipe or formula is accompanied by substantial literary expression in the form of an explanation or directions, or when there is a combination of recipes, as in a cookbook, there may be a basis for copyright protection.”
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28571692&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>But that copyright protection only extends to the literary expression. The recipe itself is still not covered by copyright, even if accompanied by an essay.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554526&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>That's not really for SEO, which favors readily accessible information.<p>That's ads. When mobile users have to scroll past 10 add, theyll click on some of them and make the blog money.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28558925&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Searching for ‘chocolate’ on this search engine turned up a surprisingly large amount of chocolate based recipes.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554845&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt;&gt; This sort of optimization is why simple recipes are typically found at the end of a rambling pointless blog post now.<p>I continue to be curious about this kind of complaint. If all you want is a recipe list, without any of the fluff, why would you click on a link to a blog, rather than on a link to a recipe aggregator?</p><p>Foodie blogs exist specifically for the people who want a foodie discussion and not just an ingredients' list.</p><p>Is it because blogs tend to have better recipes overall? In that case, isn't there a bit of entitlement involved in asking that the author self-sacrificingly provides only the information that you want, without taking care of their own needs and wants, also?
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555343&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I think the complaint is that those blogs rank higher than nuts-and-bolts recipes now. It wasn't that way a few years ago. Yes, scrolling down the results to Food Network or Martha Stewart or whatever is possible, as is going directly to those sites and using their site search, but it's noticeable and annoying.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28561850&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Not my experience. For a very quick test, I searched DDG for "omelette recipe, "carbonara recipe" and "peking duck recipe" (just to spice it up a bit) and all my top results are aggregators. Even "avgolemeono recipe" (which I'd think is very specialised) is aggregators on top.<p>To be honest, I don't follow recipes when I cook unless it's a dish I've never had before. At that point what I want is to understand the point of the dish. A list of ingredients and preparation instructions don't tell me what it's supposed to taste and smell like. The foodie blogs at least try to create a certain... feeling of place, I suppose, some kind of impression that guides you when you cook. I wouldn't say it always works but I appreciate the effort.</p><p>My real complaint with recipe writers is that they know how to cook one or two dishes well and they crib the rest off each other so even with all the information they provide, you still can't reliably cook a good meal from a recipe unless you've had the dish before. But that's my personal opinion.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555058&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>It's the same thing that people always complain about. This thing is not in a format that I like, so it must be not what anyone likes.<p>If you want JUST recipes, pay money instead of just randomly googling around. America's test kitchen has a billion, vetted, and really good recipes. That solves that problem.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28564255&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Specialization mostly a problem in monocultures.<p>If you almost only plant wheat, you are going to end up with one hell of a pest problem.</p><p>If you almost only have Windows XP, you are going to have one hell of a virus problem.</p><p>If you almost only have SearchRank-style search engines (or just the one), you are going to have one hell of a content spam problem.</p><p>Even though they have some pretty dodgy incentives, I don't think google suffers quality problems because they are evil, I think ultimately they suffer because they're so dominant. Whatever they do, the spammers adapt almost instantly.</p><p>A diverse ecosystem on the other hand limits the viability of specialization by its very nature. If one actor is attacked, it shrinks and that reduces the opportunity for attacking it.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552836&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I don't think the existing media-heavy websites are gaming Google to rank higher. It's that Google itself prefers media heavy content; they don't have to "game" anything.<p>I also think a search engine like this would be quite hard to game. An ML-based classifier trained on thousands of text-heavy and media-heavy screenshots should be quite robust and I think would be very hard to evade, so the "game" will become more about how <i>identify</i> the crawler so you can serve it a high-ranking page while serving crap to the real users, and it seems fairly easily to defeat if the search engine does a second pass using residential proxies and standard browser user agents to detect this behavior (it could also threaten huge penalties like the entire domain being banned for a month to even deter attempts at this).
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28556595&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>With the advances in text generation by machines that looks, but isn't <i>quite</i> accurate (aka GPT-3), seems like it would be <i>easily</i> gamed (given access to GPT-3). Even without GPT-3, if the content being prioritized is mere text, I'm sure that for a pile of money, I could generate something that looks like Wikipedia, in the sense that it's a giant pile of mostly text, but it would make zero sense to a human reader. (Building an SEO farm to boost ranking of not-wikpedia is left as an exercise for the reader.)</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552885&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>If there were a wider variety of popular search engines, with different ranking criteria, would sites begin to move away from gaming the system? Surely it would be too hard to game more than one search engine at a time?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28556752&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>It would be a matter of numbers anyway about which they optimize for. A/B testing is already in place and doesn't care about where it comes from, just which one does better.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563937&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>There should be some perfect balance where this search engine is N% as popular as Google, where Google soaks up all of the gamifiers, but this search engine is still popular enough to derive revenue and do ML and other search-engine-useful stuff.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553669&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; the people who want to game it will appear.<p>So just add human review to the mix, if a site is obviously trying to game the system (listicles, seo spam etc) just drop and ban them from the search index.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28562230&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt;followed by a listicle "8 Reasons Why Rome Fell"<p>but arent you curious about the 7th reason? it will surprise you!
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28560404&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Search engines whose revenue is based on advertising will ultimately be tuned to steer you to the ad foodchain. All the incentives are aligned towards and all the metrics ultimately in service of, profit for advertisers. Not in the 99% of people who can convinced to consume something by ads? Welp, screw you.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563922&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Search engines should be something you pay for. Surely search engine powerusers can afford to pay for such a service. If Google makes $1 per user per month or something, that's not too high a bar to get over.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28565299&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Search engines should be like libraries. At least some tiny sliver of the billions we spend on education and research should go to, you know, actually organizing the world's information and making it universally available.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28594439&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I see another issue here: companies like Google prioritize information to 1) keep their users and 2) maximize their profit.<p>If you move data organization to another type of organization (non-profit, state, universities - private or public), then the question of data prioritization becomes highly political. What should be exposed? What should not? What to put first? ...</p><p>It is already, but to a smaller extend since money-making companies have little interest in data meaning, and high interest in the commercial value of their users.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28574797&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>The theoretical cap for this, if you include every human being on planet Earth, is 7 billion/month. This translates into $84 billion annual revenue.<p>Google's revenue last year was 146 billion, and it operates not anywhere near the theoretical maximum. Most of that revenue is advertisement.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28556809&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>The Wikipedia link at the top is always given. It would maybe be good to make it a little clearer that it's not one of the true results.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563048&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I think this is just because of terms you have searched. In my test-searches Wikipedia has not come up once in first position (i think the highest was 3rd in the list).<p>Here's what I've tried with a few variations:
golang generics proposal,
machine learning transformer,
covid hospitalization germany</p><p>[edit] formatting
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563294&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I think maybe it's a special insert at the top, but only if a Wikipedia page is found that matches you search term? I'm not sure now though.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553256&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Or you could just search for 'rome movie'. Though for more complex disambiguation you would need to resort to, e.g. schema.org descriptions (which are supported by most search engines, and the foundation for most "smart" search result snippets).</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28562001&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>That's a fair point. This engine would be useful if you need grep over internet (by without regexes), i.e. when you want to find the exact phrases. But that's a relatively narrow use case.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553251&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I tend to prefer Wikipedia for movies. The exception is actor headshots if I'm trying to identify someone, which Wikipedia lacks for licensing reasons, but otherwise Wikipedia tends to be better than IMDB for most needs. Wikipedia has an IMDB link on every article anyway.<p>Another need I guess might be reviews, for which RT or MC are better than IMDB: not sure if either of those two will fare better than IMDB in this search engine but again Wiki has links out (in addition to good reception summaries)
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554517&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>For me, imdb was much better when they had user comments/discussion.<p>I never even posted on it myself, but browsing the discussions one could learn all sorts of trivia, inside info, speculation, etc about each movie.</p><p>Since they (inexplicably) killed that feature, I rarely even visit anymore. Your right, for many purposes wikipedia is better, especially for TV series episode lists with summaries.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554616&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>IMDB management thought it was their brilliant editorial work that drew people to their site. Morons. It was the comments all along. Of course they also believed they could create gravity-free zones by sheer force of executive will (and maybe still do).</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28562865&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Especially for old and lesser known movies, the discussion board for the movie was a brilliant addition that could give the movie an extra dimension. Context is very important in order to understand, and ulitmately enjoy something.<p>I think they removed it in part because new movies, like star wars and superhero movies, had alot of negative activity.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28560287&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I find IMDb to be more convenient than RT/MC/Wikipedia for finding release dates of movies - nearly every other website lists only the American release date, maybe one or two others if the movie was disproportionately popular in certain regions.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28566154&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Wow I used "personality test" and actually got useful articles about personality theory. I'll actually use this!</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28574515&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I think it's a case where systems diversity can be an advantage. Much like how most malware was historically written for Windows and could be avoided by using Linux, the low-quality search engine bait is created for Google and can be avoided by using a different style of search engine.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28566556&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I tried some queries for Harry Potter fanfictions, and the results were pretty much completely unrelated. There weren’t that many results, either.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28591124&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Yeah, that's just not a type of query my search engine is particularly good at. It's pretty dumb, and just tries to match as much of the webpage against the query as it can.<p>This used to be how all search engines worked, but I guess people have been taught by google that they should ask questions now, instead of search for terms.</p><p>I wonder how I can guide people to make more suitable queries. Maybe I should just make it look less like google.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552908&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I had the exact opposite experience. I searched the site for "java", got a Wikipedia link first (for the island, not the programming language), and the 2nd result was to a random JEP page, and all the rest of the results were random tidbits about Java (e.g. "XZ compression algorithm in Java). Didn't get any high level results pointing to an overview of the language, getting started guides, etc.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28555689&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>well, they're results to java related items...<p>What kind of links where you expecting to find?
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28568873&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>If this search engine ever takes off, the listicle writers will just start optimizing for it too, right?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28570409&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>If the goal was to remove modern web design, ok sure mission accomplished.<p>If your goal was to create a search engine that ignored listicles and other fluff and instead got you meatier results like "academic talks" and such, then no.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552449&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I did a search for "George Washington"<p>First result after Wikipedia:</p><p>"Radiophone Transmitter on the U.S.S. George Washington (1920)</p><p>In 1906, Reginald Fessenden contracted with General Electric to build the first alternator transmitter. G.E. continued to perfect alternator transmitter design, and at the time of this report, the Navy was operating one of G.E.'s 200 kilowatt alternators
<a href="http://earlyradiohistory.us/1919wsh.htm">http://earlyradiohistory.us/1919wsh.htm</a>
"</p><p>Another result in the first few:</p><p>" - VANDERBILT, GEORGE WASHINGTON</p><p>PH: (800) ###-#233 FX: (#03) 641-5###.
<a href="https://www.scottwinslow.com/manufacturer/VANDERBILT_GEORGE_WASHINGTON/2182">https://www.ScottWinslow.com/manufacturer/VANDERBILT_GEORGE_...</a>
"</p><p>And just below that terrible result:</p><p>"I Looked and I Listened -- George Washington Hill extract (1954)</p><p>Although the events described in this account are undated, they appear to have occurred in late 1928. I Looked and I Listened, Ben Gross, 1954, pages 104-105: Programs such as these called for the expenditure of larger sums than NBC had anticipated. It be <a href="http://earlyradiohistory.us/1954ayl2.htm">http://earlyradiohistory.us/1954ayl2.htm</a>
"</p><p>Dramatically worse than Google.</p><p>---</p><p>Ok, how about a search for "Rome" then? Surely it'll pull some great text results for the city or the ancient empire.</p><p>First result after Wikipedia:</p><p>"Home | Rome Daily Sentinel</p><p>Reliable Community News for Oneida, Madison and Lewis County
<a href="http://romesentinel.com/">http://romesentinel.com/</a>"</p><p>The fourth result for searching "Rome":</p><p>"Glenn's Pens - Stores of Note</p><p>Glenn's Pens, web site about pens, inks, stores, companies - the pleasure of owning and using a pen of choice. Direcdtory of pen stores in Europe.
<a href="http://www.marcuslink.com/pens/storesofnote/roma.html">http://www.marcuslink.com/pens/storesofnote/roma.html</a>"</p><p>Again, dramatically worse than Google.</p><p>---</p><p>Ok, how about if I search for "British"?</p><p>First result after Wikipedia:</p><p>"BRITISH MINING DATABASE</p><p>British_Mining_Database
<a href="http://www.users.globalnet.co.uk/~lizcolin/bmd.htm">http://www.users.globalnet.co.uk/~lizcolin/bmd.htm</a>
"</p><p>And after that:</p><p>"British Virgin Islands</p><p>Many of these photos were taken on board the Spirit of Massachusetts. The sailing trip was organized by Toto Tours. Images Copyright © Lowell Greenberg Home Up Spring Quail Gardens Forest Home Lake Hodges Cape Falcon Cape Lookout, Oregon Wahkeena
<a href="http://www.earthrenewal.org/british_virgin_islands2.htm">http://www.earthrenewal.org/british_virgin_islands2.htm</a>"</p><p>Again, far off the mark and dramatically worse than Google.</p><p>I like the idea of Google having lots of search competition, this isn't there yet (and I wouldn't expect it to be). I don't think overhyping its results does it any favors.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553320&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>What were you expecting to see for British? There must be millions of pages containing that term. Anyway the first screenful from Google is unadulterated crap, advertising mixed with the usual trivia questions.<p>If you are going top claim something is wide of the mark then you really ought to tell us at least roughly where the mark is.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552969&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>This is not a Google competitor, it's a different type of search engine with different goals.<p>&gt; If you are looking for fact, this is almost certainly the wrong tool. If you are looking for serendipity, you're on the right track. When was the last time you just stumbled onto something interesting, by the way?
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552594&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I checked the results of the same query and they seem fine. Lots of speeches and articles about George Washington the US president. There's even his beer recipe.<p>As for the results you linked, it's part of the zeitgeist to list other entities sharing the same name. Sure, they could use some subtle changes in ranking, but overall the returned links satisfy my curiosity.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28551057&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554298&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>It's very rare that I see a project on HN I can see myself using. This is one. Like others have said, the results can be a little rough. But they're rough in a way I think is much more manageable than the idiosynchrosies of more 'clever' search engines.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28558784&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I think you need to approach it more like grep than google. It's a forgotten art, dealing with this type of dumb search engine.<p>Like if you search for "How do I make a steak", you aren't going to get very good results. But a better query is "Steak Recipe", as that is at least a conceivable H1-tag.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28564884&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I searched like this all my life and always got expected results.<p>But just a week ago I found out that these "how", "what" questions give better and faster results on Google.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28570896&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>That switch happened some years ago. I've been unlearning and relearning how to use google for what feels like at least three or four years now.<p>The main pain-point, though, is that a lot of long-tail searches you could've used to find different results in years past, now seem to funnel you to the same set of results based on your apparent intent. At least, it has felt that way -- I'm not entirely sure how the modern google algorithm works.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28590804&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I realized this a few years ago when I observed my wife find things faster on Google than me.<p>I appreciate that it is easier for newcomers but I still hate it personally after years and especially that they cannot even avoid meddling with my queries even when I try to accept the new system and use the verbatim option.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28583558&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Try your old Google Fu skills on DuckDuckGo (or Bing I guess). I've found it to have good results anyway</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28583627&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; I think you need to approach it more like grep than google. It's a forgotten art<p>A search engine that accepted regex as the search parameter would be amazing.</p><p>I actually used this method as a field filter for a bunch of simple internal tools to search for info. Originally people were asking for individual search capabilities, but I didn't want it to become a giant project with me as the implementor of everyone's unique search capability feature request - so I just gave them regex, encoded inputs into the URL query string so they can save searches - gave em a bunch of examples to get going and now people are slowly learning regex and coming up with their own "new features" :P</p><p>But this made sense because it's a relatively small amount of data, so small that it's searched in the front end which is why it's more of a filter... I don't think pure regex would scale when used as a query on a massive DB, it would need some kind of hierachy still to only bother parsing a subset of relevant text... unless there is some clever functional regex caching algorithm that can be used.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28564366&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>So, you are re-implementing Altavista, Lycos and other old search engines.<p>They used the naive approach: you searched for "steak", and they would bring the pages which included the word "steak".</p><p>The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks.</p><p>Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page.</p><p>The rest is history.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28566250&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Effective Google search is also history.<p>I understand they are trying to maximize ad revenue and search does work very well for people who are looking for products or services.</p><p>But it no longer works well for finding information that is even slightly obscure.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28569020&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks.<p>I don't see a lot of people investing in SEO to boost their Marginalia results.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28590828&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page.<p>Then people fooled Google into showing the White House as top result when searching for "a miserable failure".</p><p>At the moment marginalia's approach of sorting pages into quality buckets based on lack of JS seems to be working extremely well, but of course it will be gamed if it gets popular.</p><p>However, I'd rather want SEO-crafting to consider itself with minimizing JS, rather than spamming links into every comment field on every blog across the globe ;-)
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28551546&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Love it, kudos! This is great for developers and others who Just Need Answers and not shopping or entertainment.<p>If you're looking for feedback, both from a UI design and utility standpoint, you might consider "inlining" results from selected sites, e.g. Wikipedia, stacked change, etc. Having worked on search for a long time, inlining (onebox etc) is a big reason users choose Google, and that channelers fail to get traction. If you're Serious(tm), dog into the publisher structure formats and format those, create a test suite, etc.</p><p>A word of caution: if this takes off, as a business it's vulnerable to Google shifting its algorithms slightly to identify the segment of users+queries who prefer these results and give the same results to those queries.</p><p>Hope this helps!
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28551665&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>If Google starts showing interesting text-heavy links instead of vapid listicles and storefronts, I have accomplished everything I ever could dream of.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28574244&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>It was a joke. The joke being that Google should launch at new product called Google Info, that would actually give you information when you search.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28551534&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Very cool project! How many websites do you have in your index? And how did you go about building it?<p>I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28588007&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>About 21 million, and I'm crawling myself.<p>Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text.</p><p>I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28603540&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>If you're open to sharing your index I could make a classifier for you, I do this for a living. 
It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553227&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>This is awesome. I've been looking for a long time for a search engine that basically takes everything Google does and does the opposite. Thank you for doing this, I will definitely be bookmarking it.<p>Is there a way to suggest or add sites? I went looking for woodgears.ca and only got one result. I also think my personal blog would be a good candidate for being indexed here but I couldn't find any results for it.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28583550&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I would also love to add some sites which I were missing...<p>Unfortunately this doesn't seem to be a feature which new search engines are focusing on - Brave Search also misses that feature...
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28560119&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>This has amazing potential. I'd encourage you to form a non-profit, turn this into something that can last as an organization without becoming what you're trying to avoid becoming. This is a good enough start that I bet you could raise a sizeable startup fund very soon from a combination of crowdfunding and foundation grants—I bet the Sloan Foundation would love this!</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28557428&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>This is absolutely wonderful. I am LOVING the results I'm getting back from it: the sort of content-rich sites that have become nigh unreachable using traditional search engines. Thank you for building this!</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28551527&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I love this, and I love (many of) the results so far! What I can't find on the site is detail about what "too many modern web design features" means. Is it just penalizing sites with tons of JavaScript?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28551727&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Javascript tags are penalized the hardest, but it also takes into consideration density of text per HTML. There's also some adjustments based on text length, which words occur in the page, etc.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28551455&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke.<p>A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays.</p><p>So when you search for two words, it translates them into IDs using the special hash map, goes to the words file and finds the least common of the words; starts with that.</p><p>Then it goes to the words file and looks up the URL range of the first word.</p><p>Then it goes to the words file and looks up the URL range of the second word.</p><p>Then it goes through the less common word's range and does a binary search for each of those in the range of the more common word.</p><p>Then it grabs the first N results, and translates them into URLs (through mariadb); and that's your search result.</p><p>I'm skipping over a few steps, but that's the very crudest of outlines.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552491&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Good stuff. I've also been toying with doing some homegrown search engine indexing (as an exercise in scalable systems), and this is a fantastic result and great inspiration.<p>Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28553192&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Well just crunching the numbers should indicate what is possible and what isn't.<p>For the moment I have just south of 20 million URLs indexed.</p><p>1 x 20 million bytes = 20 Mb.</p><p>10 x 20 million bytes = 200 Mb.</p><p>100 x 20 million bytes = 2 Gb.</p><p>1,000 x 20 million bytes = 20 Gb.</p><p>10,000 x 20 million bytes = 200 Gb.</p><p>100,000 x 20 million bytes = 2 Tb.</p><p>1,000,000 x 20 million bytes = 20 Tb.</p><p>This is still within what consumer hardware can deal with. It's getting expensive, but you don't need a datacenter to store 20 Tb worth of data.</p><p>How many bytes do you need, per document, for an index? Do you need 1 Mb of data to store index information about a page that, in terms of text alone, is perhaps 10 Kb?
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28580444&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>What crawler are you using and what kind of crawling speeds are you achieving?<p>How do you rank the results (is it based on content only) or you have external factors too?</p><p>What is your personal preferred search option of the 7 and why?</p><p>Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28584037&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; What crawler are you using and what kind of crawling speeds are you achieving?<p>Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper.</p><p>&gt;How do you rank the results (is it based on content only) or you have external factors too?</p><p>I rank based on a pretty large number of factors, incoming links weighted by the "textiness" of the source domain, and similarity to the query.</p><p>&gt; What is your personal preferred search option of the 7 and why?</p><p>I honestly use Google for a lot. My search engine isn't meant as a replacement, but a complement.</p><p>&gt; Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.</p><p>Are you kidding? I think the Patreon is a resounding success! I'm still a bit stunned. I've gotten more support and praise, not just in terms of money but also emails and comments here than I could have ever dreamed possible.</p><p>And this is just the start, too. I only recently got the search engine working this well. I have no doubt it can get much better. The fact that I have 11 people with me on that journey, even if they "just" pay my power bill, that's amazing.</p><p>I'm honestly a bit at a loss for words.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28584885&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>You have a great attitude!<p>And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it?</p><p>And what are the barriers to use something like this? You say yourself that you are using Google most of the time. Is jumping to check results on this engine going to be too much friction for most uses?</p><p>Can something like this exist in isolation? What kind of value would it need to provide for users to remember using it en-masse as an additional/primary vertical search like they do for Amazon?</p><p>Just thinking out-loud as I am also interested in the space (through <a href="http://teclis.com/">http://teclis.com</a>).
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28586065&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I think in part it may just be because I'm not trying to found a start-up, and I'm not trying to get rich quick. If I were, I would have dealt with this very differently. My integrity and the integrity of this project is far more important than my bank balance. Not everyone feels that way, and I can respect that, but I do.<p>Ultimately I think running something like this for profit would create really unhealthy incentives to make my search engine worse. Any value it brings, right now, it brings because it isn't trying to cater to every taste and every use case.</p><p>I also hate the constant "don't forget to slap the like and subscribe buttons"-shout outs of modern social media, even though I'm aware they it is extremely efficient. If I went down that route, I would become part of the problem I'm trying to help cure. I do feel the sirens' call though, it's intoxicating getting this sort of praise and attention.</p><p>I want this to be a long-term project, not a some overnight cinderella story.</p><p>In the end, my search engine is never going to replace google. It isn't trying to, it's trying to complement it. It's barely able now, but hopefully I can make it much better in the months and years to come.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28586368&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I think it's good not to have to depend on financial compensation for every single thing in your life, if you can be comfortable or do well otherwise.<p>This allows quite a bit of its own kind of freedom even if maximum financial opportunity is not fully exploited. Perhaps even because you are not grasping for every dollar on the table at all times.</p><p>You can do things without having to know if they will pay off, and if it turns out big anyway you can make money as a byproduct of what you do rather than having pure financial pursuit be the root of every goal.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28586281&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I agree with everything you say. The 'subscribe and like buttons' would not help your conversion with HN readers, on the contrary. Trying to run this for profit also would not help your conversion with this audience.<p>So given your setup is already ideal for 'conversions' for this population (low profile, high integrity, no BS) I was simply genuinely surprised that only 11 people converted given enormous visibility/interest this thread had. Hope that makes sense.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28587185&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I think it simply takes time to build trust. The threshold to sending someone money is high. I probably wouldn't send someone money based on a proof of concept and lofty ambitions alone.<p>I'd absolutely consider sending someone money if they kept bringing something of value into my life. If I want more people to join the patreon, I'll just have to earn their trust and support.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28586125&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>The day Google first appeared on the full internet it was excellent of course because it had no ads.<p>Plus another excellent feature was you would get the same search results no matter who or where you were for quite some period of calendar time.</p><p>If something new did appear it was likely to be one of the new sites that was popping up all the time and it was likely to be as worthwhile as its established associates on the front page.</p><p>You shouldn't need to crawl nearly as fast if you can compensate by treading more suitably where those have gone before.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28583136&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Interesting, in my database (<a href="http://root.rupy.se/">http://root.rupy.se</a>) I have one file per word that contains the ids (long) of the nodes (URLs), so to search many words together I have to go through the first file and one by one see if I find matches in the second.<p>How does the range binary search work, does it just prune out the overlaps, how efficient is it and how much data do you have in there for say "hello" and "world" f.ex?
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28559072&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I’m not sure how you go from word to url range? Range implies contiguous, but how can you make that happen for a bunch of words without keeping track of a list of urls for each word (or URL ids, the idea is the same)?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28562269&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>The trick is that the list of URLs for each word already <i>is</i> in the URLs file.<p>The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence.</p><p>Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8.</p><p>The first range corresponds to the URLs 1, 5, 7; and the second 2, 5, 8.</p><p>If you search for hello world, it will first pick a range, the range for "hello", let's say (1,5,7); and then do binary searches in the second range -- the range corresponding to "world" -- (2,5,8) to find the overlap.</p><p>This seems like it would be very slow, but since you can trivially find the size of the ranges, it's possible to always do them in an order of increasing range-sizes. 10 x log(100000) is a lot smaller than 100000 x log(10)
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28584703&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I'll get back to your email in a while, I've got a ton of messages I'm working through.<p>But yeah, in short pseudocode:</p><pre><code> for url in range-for-"hello":
 if binary-search (range-for-"world", url):
 yield url
</code></pre>
I do use streams, but that is the bare essence of it.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28602340&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>So every time you insert a new URL for a word you have to update the range for every other single word since the URL file will be shifted?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28608954&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>No, I actually count the n-grams as distinct words (up to 4-grams). The main limiter is for that is space, so I only extract "canned" n-grams from some tags.<p>I would first search for the bigram hello_world, that's an O(1) array lookup; as then documents merely containing the words hello and world (usually not a good search result), that's the algorithm I'm describing in the parent comment.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28609325&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Makes sense. Every time you insert a new URL for a word you have to update the ranges for every other word since the URL file will be shifted?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28577932&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I continue to want a communal set of 50-100 million urls and data that are "good" (i.e. for any value of good and more complete than common crawl) that are accessible enough to be easy to work with that can be used to experiment with different web search tech. There are enough separate problems to tackle in web search that breaking it down would maybe move the needle. We have lots of kaggle competitions about ranking, but using closed data. What other types of kaggle projects would help web search?<p>What can we do to foster a sustainable bazaar of projects to make it easier to build web search engines?
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28554927&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Not OP but crawling is easy if you don't try scanning 5+ pages a second - almost all rate limiting/heuristic based 'keep server costs low' engines, including Cloudflare, don't care if you request every page, but will take action if you do something like burst every page and take up just as many server resources as a hundred concurrent users.<p>Now, that is assuming you aren't on some VPS provider. If you're going to crawl, you'll have the best chance when you use your own IPs on your own ASN, with DNS and reverse DNS set up correctly. This makes it so the IP reputation systems can detect you as a crawler but not one that hammers every site it visits.</p><p>Also, I imagine that, for a search engine like this, it doesn't expect content to change much anyways - so it can take its time crawling every site only once every month or two, instead of the multiple times a week (or day) search engines like Google have to for the constantly-updated content being churned out.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28552925&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Pretty neat!!!<p>You may already be aware of this, but the page doesn't seem to be formatted correctly on mobile. The content shows in a single thin column in the middle.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28574545&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Really like it, and love that you've done this yourself.<p>I'd prefer if it does just one thing and does that really well. Don't waste your time on calculator and conversion functions, or pseudo-natural language queries. There are plenty of good calculator &amp; converter tools and websites, but we all need a really good search engine. I think you'd be better looking at handling synonyms and plurals.</p><p>Thanks.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28578625&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I've mostly added these converters and frills because I'm trying to eat my own dogfood, and google is deeply engrained as my go-to source for unit conversions and back-of-envelope calculations.<p>Don't worry, this stuff is easy, and doesn't even remotely take away from the work on the harder problems.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28558891&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Awesome project! How are you able to keep the site running after HN kiss of death? What is your stack, elastic search or something simper? How did you crawl so many websites for a project this size? Did you use any APIs like duck duck go or data from other search engines? Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28563416&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>&gt; How are you able to keep the site running after HN kiss of death?<p>I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware.</p><p>My day job is also developing this sort of highly performance java applications, I guess it helps.</p><p>&gt; What is your stack, elastic search or something simper?</p><p>It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries.</p><p>&gt; How did you crawl so many websites for a project this size?</p><p>It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains).</p><p>&gt; Did you use any APIs like duck duck go or data from other search engines?</p><p>Nope, it's all me.</p><p>&gt; Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?</p><p>I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains.</p><p>It would be interesting to try a page rank-style approach, but my thinking is that because it's <i>the</i> algorithm, it's also the algorithm everyone is trying to game.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28562700&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Thank you so much for creating such a useful search engine!<p>Is there any way that you can get an HTTP certificate?</p><p>I use an old iPhone 4S, and most of the modern web is inaccessible due to TLS. Hacker News and mbasic.facebook are two of the last sites I can use.</p><p>Usually text-based sites are more accessible, so this could be really useful to help me continue using my antique devices!
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28560568&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>Love the idea. A little feedback: layout needs tweaking for mobile. FWIW: I'm on mobile Firefox for Android.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28578085&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>This is really cool! A spacer between the links would help my old eyes; I keep getting lost in which link goes with which description. :-)</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28551308&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>Hi,<p>Interesting idea. Definitely see an overlap with eReader markets and looking at text only contents.</p><p>How does it work?</p><p>It ignores pages on which it detects frameworks for ui and ads or any javascript code at all?
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28573058&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I can't see the letters in the disturbing white search box, I'm on duckduckgo, brave, monocles, jquarks, smartcookieweb on Android.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28560865&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>What was the starting site you fed to the crawler to follow the links from to build the index?</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28578649&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>It's custom java code, for the most part. I'm using mariadb for some ancillary information, but the index and the crawler and everything is written from scratch.<p>It's hosted on my consumer-equipment server (Ryzen 3900x, 128Gb ram, Optane 900p+a few IronWolf drives), bare bones on Debian.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28557616&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><p>I love this idea, and admire the work you put into it. I'm a fan of long reads and historical non-fiction, and Google's results are truly garbage.<p>I have a criticism that I think may pertain to the ranking methodology. I searched for "discovery of Australia". Among the top results were:</p><p>* A site claiming that the biblical flood was caused by Earth colliding with a comet (with several other pages from that site also making the top search results with other wild claims, e.g. that the Egyptians discovered Arizona);</p><p>* Another site claiming the first inhabitants of Australia were a lost tribe of Israel;</p><p>* A third site claiming that Australia was discovered and founded by members of a secret society of Rosicrucians who had infiltrated the Dutch East India Company and planned to build an Australian utopia...</p><p>These were all pages heavy with HTML4 tags and virtually devoid of Javascript, the kinds of pages you'd frequently see in the late 1990s from people who had built their own static websites in a text editor, or exported HTML from MS Word. At that time, there were millions of those sites with people paying for their own unique domain names, and so the proportion of them that were home to wild-eyed conspiracy theories was relatively small. What I think has happened is that kooks continued to keep these sites up - to the point where it's almost a visual trope now to see a red &lt;h1&gt; tag in Times New Roman and think, uh oh, I've stumbled on an "ancient aliens" site. Whereas scholars and journals offering higher quality information have moved to more modern platforms that rely more heavily on modern browsers - with or without their own domain names. So as a result what seemed to surface here were the fragments of the old web that remain live - possibly because people living in cabins in Montana forget to cancel their web hosting, or because the nature of old-school conspiracy theorists is to just keep packing their old sites with walls of text surrounded by &lt;p&gt; tags.</p><p>Arguably, this seems to rank the way Google's engine used to, since it couldn't run JS and they wanted to punish sites that used code to change markup at render time. At least, when I used to have to do onsite SEO work, it was always about simple tag hierarchies.</p><p>I wonder whether there isn't some better metric of validity and information quality than what markup is used. Some of the sites that surfaced further down could be considered interesting and valuable resources. I think <i>not punishing</i> simple wall-of-text content is a good thing. But to punish more complicated layouts may have the perverse effect of downranking higher-quality sources of information - i.e. people and organizations who can afford to build a decent website, or who care to migrate to a modern blogging platform.
 </p></p></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td><table> <tbody><tr> <td></td><td>
 <a href="vote?id=28575222&amp;how=up&amp;goto=item%3Fid%3D28550764"></a> </td><td><br /><div>
 <p>I don’t want my search engine to somehow try to judge the believability of the results. I’d like to be the judge of that myself.</p></div></td></tr>
 </tbody></table></td></tr>
 <tr><td></td></tr>
 <tr><td></td></tr>
 <tr></tr><tr><td></td></tr>
 </tbody></table>
</td></div>||||I|||| Hacker News new | past | comments | ask | show | jobs | submit login                  
                   

  A search engine that favors text-heavy sites and punishes modern web design ( marginalia.nu )                
 3441 points by Funes- on Sept 16, 2021 | hide | past | favorite | 717 comments                  
                   
                   
                   
 snakeboy on Sept 16, 2021 | next [–]                  
                   
 Wow, that's awesome. Great work!                  
                   
 For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources.                
                   
 When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totally useless.                
                   
                   
 foofoo4u on Sept 16, 2021 | parent | next [–]                  
                   
 Good comparison. Reminds me of an analogy I like to make of today's web, which is it feels like browsing through a magazine store — full of top 10s, shallow wow-factoids, and baity material. I genuinely believe terrible results like this are making society dumber.               
                   
                   
 rchaud on Sept 16, 2021 | root | parent | next [–]                  
                   
 The context matters. I'd happily read "Top 10" lists on a website if the site itself was dedicated to that one thing. "Top 10 Prog Rock albums", while a lazy, SEO-bait title, would at least be credible if it were on a music-oriented website.               
                   
 But no, these stories all come from cookie-cutter "new media" blog sites, written by an anonymous content writer who's repackaged Wikipedia/Discogs info into Buzzfeed-style copy writing designed to get people to "share to Twitter/FB". No passion, no expertise. Just eyeballs at any cost.              
                   
                   
 foofoo4u on Sept 16, 2021 | root | parent | next [–]                  
                   
 This got me thinking that maybe one of the other big reasons for this is that the algorithms prioritize newer pages over older pages. This produces the problem where instead of covering a topic and refining it over time, the incentive is to repackage it over and over again.               
                   
 It reminds me of an annoyance I have with the Kindle store. If I wanted to find a book on, let's say, Psychology, there is no option to find all-time respected books of the past centenary. Amazon's algorithms constantly push to recommend the latest hot book of the year. But I don't want that. A year is not enough time to have society determine if the material withstands time. I want something that has stood the test of time and is recommended by reputable institutions.           
                   
                   
 jamra on Sept 17, 2021 | root | parent | next [–]                  
                   
 This is just a guess, but I believe that they use machine learning and rank it by the clicks. I took some coursera courses and Andrew Ng sort of suggested that as their strategy.                
                   
 The problem is that clickbait and low effort articles could be good enough to get the click, but low effort enough to drag society into the gutter. As time passes, the system is gamified more and more where the least effort for the most clicks is optimized.               
                   
                   
 HPsquared on Sept 18, 2021 | root | parent | next [–]                  
                   
 It sounds like the problem is, the search engine has no way to measure satisfaction after the click.                 
                   
                   
 sdoering on Sept 18, 2021 | root | parent | next [–]                  
                   
 But they have. or could have. At least Google (and to a smaller extend Microsoft), if you are using Chrome/Bing have exactly that signal. If you stay on the site and scroll (taking time, reading, not skimming) all this could be a signal to evaluate if the search result met your needs.              
                   
                   
 digaozao on Sept 18, 2021 | root | parent | prev | next [–]                  
                   
 I've heard google would guess with bounce rate. Or another way, if the user clicks on LinkedIn website A, after a few moments keeps trying other linksw/related search. It would mean it was not valuable.                
                   
                   
 dtonon on Sept 18, 2021 | root | parent | prev | next [–]                  
                   
 They tried to insight this with the "bounce rate"                  
                   
                   
 amenod on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 > is that the algorithms prioritize newer pages over older pages.                  
                   
 They do? That would explain a lot - but ironically, I can't find a good source on this. Do you have one at hand?                 
                   
                   
 dvogel on Sept 16, 2021 | root | parent | next [–]                  
                   
 It is pretty obvious if you search for any old topic that is also covered incessantly by the news. "royal family" is a good example. There's no way those news stories published an hour ago are listed first due to a high PageRank score (which necessarily depends on time to accumulate inbound links).              
                   
                   
 RattleyCooper on Sept 16, 2021 | root | parent | next [–]                  
                   
 It depends on the content. The flip side is looking up a programming-related question and getting results from 2012.                 
                   
 I think they take different things into account based on the thing being searched.                  
                   
                   
 II2II on Sept 17, 2021 | root | parent | next [–]                  
                   
 Even your example would depend upon the context. There are many cases where a programming question in 2021 is identical to one from 2012, along with the answer. In those instances, would you rather a shallow answer from 2021 or an indepth answer from 2012? This is not meant to imply that older answers offer greater depth, yet a heavy bias towards recent material can produce that outcome in some circumstances.            
                   
                   
 valvar on Sept 17, 2021 | root | parent | next [–]                  
                   
 If you're using tools/languages that change rapidly (like Kotlin, in my case), syntax from a few years ago will often be outdated.                 
                   
                   
 II2II on Sept 17, 2021 | root | parent | next [–]                  
                   
 Yes, yet there are programming questions that go beyond "how do I do X in language Y" or "how do I do X with library Y". The language and library specific questions are the ones where I would be less inclined to want additional depth anyhow, well, provided they aren't dependent upon some language or library specific implementation detail.              
                   
                   
 post_below on Sept 18, 2021 | root | parent | prev | next [–]                  
                   
 There are of course a variety of factors, including the popularity of the site the page is published on. The signals related to the site are often as important as the content on the page itself. Even different parts of the same site can lend varying weight to something published in that section.              
                   
 Engagement, as measured in clicks and time spent on page, plays a big part.                  
                   
 But you're right, to a degree, as frequently updated pages can rank higher in many areas. A newly published page has been recently updated.                 
                   
 A lot depends on the (algorithmically perceived) topic too. Where news is concerned, you're completely right, algos are always going to favor newer content unless your search terms specify otherwise.                
                   
 PageRank, in it's original form, is long dead. Inbound link related signals are much more complex and contextual now, and other types of signals get more weight.                
                   
                   
 rchaud on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Your Google search results show the date on articles do they not? If people are more likely to click on "Celebrity Net Worth (2021)" than "Celebrity Net Worth (2012)", then the algo will update to favour those results, because people are clicking on them.               
                   
 The only definitive source on this would be the gatekeeper itself. But Google never says anything explicitly, because they don't want people gaming search rankings. Even though it happens anyway.                
                   
                   
 echelon on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 The new evergreen is refreshed sludge for bottom dollar. College kids stealing Reddit comments or moving around paragraphs from old articles. Or linking to linked blogs that link elsewhere.                
                   
 It's all stamped with Google Ads, of course, and then Google ranks these pages high enough to rake in eyeballs and ad dollars.                 
                   
 Also there's the fact that each year, the average webpage picks up two more video elements / ad players, one or two more ad overlays, a cookie banner, and half a dozen banner/interstitials. It's 3-5% content spread thinly over an ad engine.               
                   
 The Google web is about squeezing ads down your throat.                  
                   
                   
 andrepd on Sept 17, 2021 | root | parent | next [–]                  
                   
 Really makes you wonder: you play whack a mole and tackle the symptoms with initiatives like this search engine. But the root of that problem and many many others is the same: advertising. Why don't we try to tackle that?               
                   
                   
 HPsquared on Sept 18, 2021 | root | parent | next [–]                  
                   
 Perhaps a subscription-based search engine would avoid these incentives.                  
                   
                   
 QuibbleQuota on Sept 18, 2021 | root | parent | prev | next [–]                  
                   
 Let’s go a few levels deeper and question our consumption culture                  
                   
                   
 echelon on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Exactly.                   
                   
 The only reason people make content they aren't passionate about is advertising.                  
                   
                   
 hodgesrm on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 > This got me thinking that maybe one of the other big reasons for this is that the algorithms prioritize newer pages over older pages.                 
                   
 Actually that's not always the case. We publish a lot of blog content and it's really hard to publish new content that replaces old articles. We still see articles from 2017 coming up as more popular than newer, better treatments of the same subject. If somebody knows the SEO magic to get around this I'm all ears.              
                   
                   
 WalterBright on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Amazon search clearly does not prioritize exact title matches.                  
                   
                   
 1vuio0pswjnm7 on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Its the "healthy web" Mozilla^1 and Google keep telling their blog audiences about. :)                  
                   
 1 Accept quid pro quo to send all queries to Google by default                  
                   
 If what these companies were telling their readers was true, i.e., that advertising is "essential" for the web to survive, then how are the sites returned by this search engine for text-heavy websites (that are not discoverable through Google, the default search engine for Chrome, Firefox, etc.) able to remain online. Advertising is essential for the "tech" company middleman business to survive.             
                   
                   
 Dah00n on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 I'm not sure I agree with your example. It seems to me it is the exact same as a "Top ten drinks to drink on a rainy day" list. There's simply too many good albums and opinions differ, so a top ten would -just like the drinks- end up being a list of the most popular ones with maybe one the author picks to stir some controversy or discussion. In my opinion the world would be a smarter place if Google ranked all such sites low. Then we might at least get fluff like "Top ten prog rock albums if you love X, hate Y and listen to Z when no one is around" instead.          
                   
                   
 dageshi on Sept 17, 2021 | root | parent | next [–]                  
                   
 Google won't rank them low because they actually do serve an important purpose. They're there for people who don't really know what they want specifically, they're looking for an overview. A top 10 gives a digestible overview on some topic, which helps the searcher narrow down what they really want.              
                   
 A "Top 10 albums of all time" post is actually better off going through 10 genres of popular music from the past 50 years and picking the top album (plus mentioning some other top albums in the genre) for each one.                
                   
 That gives the user the overview they're probably looking for, whether those are the top 10 albums of all time or not. It's a case of what the user searched for vs what they actually really want.                
                   
                   
 bythckr on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 "The best minds of my generation are thinking about how to make people click ads"                  
                   
                   
 redwall_hp on Sept 18, 2021 | root | parent | prev | next [–]                  
                   
 So did Tim Berners-Lee. He was vehemently opposed to people shoehorning images into the WWW, because he didn't want it to turn into the equivalent of magazines. Which, I believe, he shared the opinion of them making society dumber.               
                   
 Appropriately enough, I couldn't find a good quote to verify that since Google is only giving me newspapers and magazines talking about Sir Tim in the context of current events. I do believe it's in his book "Weaving the Web" though.               
                   
                   
 wwweston on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 It's also possible that it's the other way around: a certain "common denominator" + algorithms that chase broad engagement = mediocre results.                 
                   
 The real trick would be some kind of engine that can aim just above where the user's at.                  
                   
                   
 anonuser123456 on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 > I genuinely believe terrible results like this are making society dumber.                  
                   
 You have to e causality reversed. Google results reflect the fact that society is dumb.                  
                   
                   
 jimbokun on Sept 17, 2021 | root | parent | next [–]                  
                   
 Google results reflect the fact that educating and informing people has low profit margins.                 
                   
                   
 crucialfelix on Sept 19, 2021 | root | parent | prev | next [–]                  
                   
 Or the distribution of people now online better reflects stupidity in the general population.                 
                   
                   
 bluGill on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 what I really want is a true AI to search through all that and figure out the useful truth. I don't know how to do this (and of course whoever writes the AI needs to be unbiased...)                
                   
                   
 idiotsecant on Sept 17, 2021 | root | parent | next [–]                  
                   
 >whoever writes the AI needs to be unbiased...)                  
                   
 I'm not sure the idea of a sentient being not having a bias is meaningful. Reality, once you get past the trivial bits, is subjective.                 
                   
                   
 jimbokun on Sept 17, 2021 | root | parent | next [–]                  
                   
 Isn't there a fundamental ML postulate that learning without bias is impossible?                  
                   
 Maybe not the same kind of bias we think of in terms of politics and such, but I wonder if there's a connection.                 
                   
                   
 bluGill on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 I didn't say the AI should be unbiased, just whoever writes it.                  
                   
 I want an AI that is biased to the truth when there is an objective one, and my tastes otherwise. (that is when asked to find a good book it should give me fantasy even though romance is the most popular genre and so will have better reviews)               
                   
                   
 hnick on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 I think that is the goal, it's just what we currently have is an AI that's like a naive child who is easily tricked and distracted by clickbait.                 
                   
                   
 Dah00n on Sept 17, 2021 | root | parent | next [–]                  
                   
 >an AI that's <snip> easily tricked and distracted by clickbait.                  
                   
 So, AIs are actually on par with most adults now? (Sorry)                  
                   
                   
 SPBS on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Cool, it appears that the trend towards JS may be causing self-selection -- if a page has a high amount of JS, it is highly unlikely to contain anything of value.                
                   
                   
 fullsend on Sept 16, 2021 | root | parent | next [–]                  
                   
 True. Unfortunately many large corporate websites through which you pay bills, order tickets, etc. are becoming infested with JS widgets and bulky, slow interfaces. These are hard to avoid.                
                   
                   
 artificial on Sept 17, 2021 | root | parent | next [–]                  
                   
 Conversely no software to install. Browser as a platform. Don’t have to boot to Windows to pay your bills with activex for example                 
                   
                   
 foxfluff on Sept 17, 2021 | root | parent | next [–]                  
                   
 The mostly JS-less web was fine, fast, and reliable 20 years ago and I never had ActiveX.                 
                   
 I hear stories about Flash and ActiveX but I literally never needed these to shop or pay bills online. Payments also didn't require scripts from a dozen domains and four redirects..                
                   
                   
 artificial on Sept 18, 2021 | root | parent | next [–]                  
                   
 Yup, and taking payments online was awful but privacy was more of a thing. In South Korea ActiveX was required until recently. https://www.theregister.com/2020/12/10/south_korea_activex_c...                
                   
                   
 TeMPOraL on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 The platform isn't the problem. The problem is with the amount of code that does something other than letting you "pay bills, order tickets, etc.".                 
                   
                   
 Dah00n on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Huh. A weighted algorithm, somewhere between Google and the one linked, where you could subtract from sites by amount of JavaScript might be interesting.                
                   
                   
 dv_dt on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 If one could create an metric of ad to content ratio from the js used, I would guess that would be a nice differentiator too.                 
                   
                   
 hinkley on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Browsers should be cherry picking the most compelling things that people accomplish with complex code and supporting them as a native feature. Maybe the Browser Wars aren’t keeping up anymore.                
                   
                   
 lugged on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Was that ever in doubt?                   
                   
                   
 klntsky on Sept 16, 2021 | parent | prev | next [–]                  
                   
 However, when searching for "haskell type inference algorithm" I get completely useless results.                 
                   
                   
 burkaman on Sept 16, 2021 | root | parent | next [–]                  
                   
 That query is too long apparently. But if you shorten to "haskell type inference", I think it delivers on its promise:                 
                   
 > If you are looking for fact, this is almost certainly the wrong tool. If you are looking for serendipity, you're on the right track. When was the last time you just stumbled onto something interesting, by the way?                
                   
                   
 marginalia_nu on Sept 16, 2021 | root | parent | next [–]                  
                   
 The search engine doesn't do any type of re-ordering or synonym stuff, it only tires to construct different N-grams from the search query.                 
                   
 So if you for example compare "SDL tutorial" with "SDL tutorials". On google you'd get the same stuff, this search engine, for better or worse doesn't.                 
                   
 This is a design decision, for now anyway, mostly because I'm incredibly annoyed when algorithms are second-guessing me. On the other hand, it does mean you sometimes have to try different searches to get relevant results.               
                   
                   
 leephillips on Sept 16, 2021 | root | parent | next [–]                  
                   
 I like this design decision. It pays you back for choosing your search terms carefully.                  
                   
                   
 mananaysiempre on Sept 17, 2021 | root | parent | next [–]                  
                   
 I’m not against a stemmer, actually, just against the aggressive concordances (?) that Google now employs, like when it shows me X in Banach spaces (the classical, textbook case) when I’m specifically searching for X in Fréchet spaces (the generalization I want to find but am not sure exists); of course Banach spaces and Fréchet spaces are almost exclusively encountered in the same context, but it doesn’t mean that one is a popular typo for the other! (The relative rarity of both of these in the corpus probably doesn’t help. The farcical case is BRST, or Becchi-Rouet-Stora-Tyutin, in physics, as it is literally a single key away from “best” and thus almost impossible to search for.)        
                   
 On the other hand, Google’s unawareness of (extensive and ubiquitous) Russian noun morphology is essentially what allowed Yandex to exist: both 2011 Yandex and 2021 Google are much more helpful for Russian than 2011 Google. I suspect (but have not checked) that the engine under discussion is utterly unusable for it. English (along with other Germanic and Romance languages to a lesser extent) is quite unusual in being meaningfully searchable without any understanding of morphology, globally speaking.           
                   
                   
 medstrom on Sept 17, 2021 | root | parent | next [–]                  
                   
 I thought you could fix that by enclosing "BRST" in quotes, but apparently not. DuckDuckGo (which uses Google) returns a couple of results that do contain "BRST" in a medical context, but most results don't contain this string at all. What's going on?               
                   
                   
 mananaysiempre on Sept 17, 2021 | root | parent | next [–]                  
                   
 I’m not certain what DDG actually uses (wasn’t it Bing?), but in my experience from the last couple of months it ignores quotes substantially more eagerly than Google does. For this particular term, a little bit of domain knowledge helps: even without quotes, brst becchi , brst formalism , brst quantization or perhaps bv brst will get you reasonable results. (I could swear Google corrected brst quantization to best quantization a year ago, but apparently not anymore.) Searching for stuff in the context of BRST is still somewhat unpleasant, though.          
                   
 I... don’t think anything particularly surprising is happening here, except for quotes being apparently ignored? I’ve had it explained to me that a rare word is essentially indistinguishable from a popular misspelling by NLP techniques as they currently exist, except by feeding the machine a massive dictionary (and perhaps not even then). BRST is a thing that you essentially can’t even define satisfactorily without at the very least four years of university-level physics (going by the conventional broad approach—the most direct possible road can of course be shorter if not necessarily more illuminating). “Best” is a very popular word both generally and in searches, and the R key is next to E on a Latin keyboard. If you are a perfect probabilistic reasoner with only these facts for context (and especially if you ignore case), I can very well believe that your best possible course of action is to assume a typo.    
                   
 How to permit overriding that decision (and indeed how to recognize you’ve actually made one worth worrying about without massive human input—e.g. Russian adjectives can have more than 20 distinct forms, can be made up on the spot by following productive word-formation processes, and you don’t want to learn all of the world’s languages!) is simply a very difficult problem for what is probably a marginal benefit in the grand scheme of things.            
                   
 I just dislike hitting these margins so much.                  
                   
                   
 leephillips on Sept 17, 2021 | root | parent | next [–]                  
                   
 It would not be a difficult problem if they allowed the " " operator to work as they claim it does, or revive the + operator.                 
                   
                   
 mananaysiempre on Sept 17, 2021 | root | parent | next [–]                  
                   
 In English, maybe; in Russian, I frequently find myself reaching for the nonexistent “morphology but not synonyms” operator (as the same noun phrase can take a different form depending on whether it is the subject or the object of a verb, or even on which verb it is the object of); even German should have the same problem AFAIU, if a bit milder. I don’t dare think about how speakers of agglunative languages (Finnish, Turkish, Malayalam) suffer.            
                   
 (DDG docs do say it supports +... and even +"...", but I can’t seem to get them to do what I want.)                 
                   
                   
 leephillips on Sept 17, 2021 | root | parent | next [–]                  
                   
 Ah, OK. I don’t know anything about Russian. This is a hard problem. I think the solution is something like what you suggest: more operators allowing different transformations. Even in English, I would like a "you may pluralize but nothing else" operator.               
                   
                   
 mananaysiempre on Sept 18, 2021 | root | parent | next [–]                  
                   
 Well it’s not that alien, it (along with the other Eastern Slavic languages, Ukrainian and Belarusian) is mostly a run-of-the-mill European language (unlike Finnish, Estonian or Hungarian) except it didn’t lose the Indo-European noun case system like most but instead developed even more cases. That is, where English or French would differentiate the roles of different arguments of a verb by prepositions or implicitly by position, Russian (like German and Latin) has a special axis of noun forms called “case” which it uses for that (and also prepositions, which now require a certain case as well—a noun form can’t not have a case like it can’t not have a number).        
                   
 There are six of them (nominal [subject], genitive [belonging, part, absence, “of”], dative [indirect object, recipient, “to”], accusative [direct object], instrumental [device, means, “by”], prepositional [what the hell even is this]), so you have (cases) × (numbers) = 6 × 2 = 12 noun forms, and adjectives agree in number and gender with their noun, but (unlike Romance languages) plurals don’t have gender, so you have (cases) × (numbers and genders) = 6 × (3 + 1) = 24 adjective forms.           
                   
 None of this would be particularly problematic, except these forms work like French or Spanish verbs: they are synthetic (case, number and gender are all a single fused ending, not orthogonal ones) and highly convoluted with a lot of irregularities. And nouns and adjectives are usually more important for a web search than verbs.              
                   
                   
 yesenadam on Sept 19, 2021 | root | parent | prev | next [–]                  
                   
 > BRST, or Becchi-Rouet-Stora-Tyutin is literally a single key away from “best” and thus almost impossible to search for.                 
                   
 Hmm I seem to be getting only relevant results, no "best", not sure what you mean. Are you not doing verbatim search?                 
                   
 https://www.google.com/search?q=brst&tbs=li:1                  
                   
                   
 dahauns on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 English is more the outlier in regard to Germanic languages, try German or Finnish, with their wonderful compounds :)                 
                   
 https://e.humanities.uva.nl/publications/2004/kamp_lang04.pd...                  
                   
                   
 mananaysiempre on Sept 18, 2021 | root | parent | next [–]                  
                   
 Well yeah, English is kind of weird, but Finnish isn’t a Germanic language at all? It’s not even Indo-European, so even Hindi is ostensibly closer to English than Finnish. I understand Standard German (along with Icelandic) is itself a bit atypical in that it hasn’t lost its cases when most other Germanic languages did.              
                   
 Re compounds, I expected they would be more or less easy to deal with by relatively dumb splitting, similar to greedy solutions to the “no spaces” problem of Chinese and Japanese, and your link seems to bear that out. But yeah, cheers to more language-specific stuff in your indexing. /s              
                   
                   
 dahauns on Sept 20, 2021 | root | parent | next [–]                  
                   
 Gaaah, brain fart - you're right, of course, dunno why I included it.                  
                   
                   
 ford_o on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Maybe list the synonyms under the query, so its easier to try different formulations.                  
                   
                   
 akavel on Sept 17, 2021 | root | parent | next [–]                  
                   
 Oh this sounds like it could be a really cool idea! This way it could also be subtly teaching users that the engine doesn't do automatic synonyms translation so it's worth experimenting; also kinda like giving the synonyms feature while still keeping user in full control.               
                   
                   
 OneLeggedCat on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Don't change it. It's good this way.                  
                   
                   
 Razengan on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 It could simply become an option.                  
                   
                   
 klntsky on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Since it does not use synonyms, it looks like it is unable to answer "how's that thing called"-queries.                 
                   
                   
 LanceH on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 It would be nice if we could pipe search engines.                  
                   
                   
 BenoitP on Sept 16, 2021 | root | parent | next [–]                  
                   
 Definitely; We could create a meta search engine that queries them all, in desktop application format.                 
                   
 Let's name it after a famous old scientist, and maybe add the year to prove it's modern: Galileo 2021.                 
                   
                   
 PaulHoule on Sept 16, 2021 | root | parent | next [–]                  
                   
 Meta search engines leave a bad taste in everyone's mouth because they've always failed. Here is why                 
                   
 https://en.wikipedia.org/wiki/Arrow%27s_impossibility_theore...                  
                   
 You can't combine a few different ranked lists and expect to get results better than any of the original ranked lists.                 
                   
                   
 robrenaud on Sept 16, 2021 | root | parent | next [–]                  
                   
 > You can't combine a few different ranked lists and expect to get results better than any of the original ranked lists.                 
                   
 I am skeptical of this application of the theorem. Here is my proposal:                  
                   
 Take the top 10 Google and Bing results. If the top result from Bing is in the top 10 from Google, display Google results. If the top result from Bing is not in the top 10 from Google, place it at the 10th position. You'd have an algorithm that ties with Google, say 98% of the time, beats it say, 1.2% of the time, and loses .8% of the time.              
                   
                   
 vikingerik on Sept 16, 2021 | root | parent | next [–]                  
                   
 Right. Arrow's theorem just says it's impossible to do it in all cases. It's still quite possible to get an improvement in a large proportion of cases, as you're proposing.                
                   
                   
 PaulHoule on Sept 16, 2021 | root | parent | next [–]                  
                   
 I've had jobs tuning up the relevance of search engines with methods like                  
                   
 https://ccc.inaoep.mx/~villasen/bib/AN%20OVERVIEW%20OF%20EVA...                  
                   
 and the first conclusion is "something that you think will improve relevance probably won't"; the TREC conference went for about five years before making the first real discovery                
                   
 https://en.wikipedia.org/wiki/Okapi_BM25                  
                   
 It's true that Arrow's Theorem doesn't strictly apply, but thinking about it makes it clear that the aggregation problem is ill-defined and tricky. (e.g. note also that a ranking function for full text search might have a range of 0-1 but is not a meaningful number, like a probability estimate that a document is relevant, but it just means that a result with a higher score is likely to be more relevant than one with a lower score.)            
                   
 Another way to think about it is that for any given feature architecture (say "bag of words") there is an (unknown) ideal ranking function.                 
                   
 You might think that a real ranking function is the ideal ranking function plus an error and that averaging several ranking functions would keep the contribution of the ideal ranking function and the errors would average out, but actually the errors are correlated.               
                   
 In the case of BM25 for instance, it turns out you have to carefully tune between the biases of "long documents get more hits because they have more words in them" and "short documents rank higher because the document vectors are spiky like the query the vectors". Until BM25 there wasn't a function that could be tuned up properly and just averaging several bad functions doesn't solve the real problem.             
                   
                   
 random314 on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Arrows theorem simply doesn't apply here. We don't need our personalized search results to satisfy the majority.                 
                   
                   
 PaulHoule on Sept 16, 2021 | root | parent | next [–]                  
                   
 But in both cases you face the problem of aggregating preferences of many into one. In one case you are combining personal preferences in the other case aggregating ‘preferences’ expressed by search engines.                
                   
                   
 random314 on Sept 17, 2021 | root | parent | next [–]                  
                   
 But search engines aren't voting to maximize the chances that their preferred candidate shows up on top. The mixed ranker has no requirement to satisfy Arrows integrity constraints. It has to satisfy the end user, which is quite possible in theory.               
                   
 Conditions the mixed ranker doesn't have to satisfy "ranking while also meeting a specified set of criteria: unrestricted domain, non-dictatorship, Pareto efficiency, and independence of irrelevant alternatives"                
                   
                   
 PaulHoule on Sept 17, 2021 | root | parent | next [–]                  
                   
 Sure, but the problem that conventional IR ranking functions are not meaningful other than by ordering leads you to the dismal world of political economy where you can't aggregate people's utility functions. (Thus you can't say anything about inequality, only about Pareto efficiency)              
                   
 Hypothetically you could treat these functions as meaningful but when you try you find that they aren't very meaningful.                 
                   
 For instance IBM Watson aggregated multiple search sources by converting all the relevance scores to "the probability that this result is relevant".                 
                   
 A conventional search engine will do horribly in that respect, you can fit a logit curve to make a probability estimator and you might get p=0.7 at the most and very rarely get that, in fact, you rarely get p>0.5.                
                   
 If you are combining search results from search engines that use similar approaches you know those p's are not independent so you can't take a large numbers of p=0.7's and turn that into a higher p.                
                   
 If you are using search engines that use radically different matching strategies (say they return only p=0.99 results with low recall) the Watson approach works, but you need a big team to develop a long tail of matching strategies.               
                   
 If you had a good p-estimator for search you could do all sorts of things that normal search engines do poorly, such as "get an email when a p>0.5 document is added to the collection."                
                   
 For now alerting features are either absent or useless and most people have no idea why.                  
                   
                   
 gnramires on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 That's an invalid application of this theorem. (It doesn't necessarily hold)                  
                   
 Suppose there's an unambiguous ranked preference by all people among a set (webpages, ranking). Suppose one search engine ranks correctly the top 5 results and incorrectly the next 5 results, while another ranks incorrectly the top 5 and correctly the next 5.               
                   
 What can happen is that some there may be no universally preferred search engine (likely). In practice, as another commenter noted, you can also have most users prefer more a certain combination of results (that's not difficult to imagine, for example by combining top independent results from different engines for example).              
                   
                   
 overkalix on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 ... is this Galileo 2021 a reference that I am not understanding?                  
                   
                   
 BenoitP on Sept 16, 2021 | root | parent | next [–]                  
                   
 Yup, but so far no one got it.                  
                   
 There was such an app in the early 2000's, before Google went mainstream, and Altavista-like engines were not good: Copernic 2000.                 
                   
 I guess I'm officially old now.                  
                   
                   
 genewitch on Sept 16, 2021 | root | parent | next [–]                  
                   
 For years I wanted to try Copernic Summarizer. It seemed like it actually worked. Then software that did summaries disappeared, maybe? And about 5 years ago bots on Reddit were doing summaries of news stories (and then links in comments).               
                   
 This is a pattern I see over and over again, some research group or academics show that something can be done (summaries that make sense and are true summaries, evolutionary algorithm FPGA programming, real time gaze prediction, etc) and there's a few published code repos and a bit of news, then 'poof' - no where to be seen for 15 years or more.             
                   
                   
 tomerv on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 FWIW, I got the reference. Maybe I'm old too?                  
                   
                   
 squeaky-clean on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 I was always a dogpile user :p                  
                   
                   
 LightG on Sept 17, 2021 | root | parent | next [–]                  
                   
 Hotbot!                   
                   
                   
 collinmanderson on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Brave browser currently has "Google fallback" which sometimes mixes in Google search results with Brave's own search engine.                 
                   
 https://search.brave.com/help/google-fallback                  
                   
                   
 gmueckl on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Not an app, but probably comes quite close in all other respects: https://metager.org                  
                   
                   
 Torwald on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 I need that with a simpler interface, so I call it after a famous dedective: Sherlock.                  
                   
                   
 artificial on Sept 16, 2021 | root | parent | next [–]                  
                   
 a magic pop sound is faintly audible as a new side project is appended to several lists Excellent, thank you!                 
                   
                   
 mkr-hn on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Likely trademark collision with this: https://www.galileo.usg.edu/                  
                   
                   
 hdjjhhvvhga on Sept 16, 2021 | parent | prev | next [–]                  
                   
 As long as few people use it, it will be great. Rest assured that the moment it becomes popular, the people who want to game it will appear.                 
                   
                   
 jandrese on Sept 16, 2021 | root | parent | next [–]                  
                   
 This sort of optimization is why simple recipes are typically found at the end of a rambling pointless blog post now.                 
                   
 Still, the best way to break SEO is to have actual competition in the search space. As long as SEO remains focused on Google there is an opportunity for these companies to thrive by evading SEO braindamage.                
                   
                   
 WorldMaker on Sept 16, 2021 | root | parent | next [–]                  
                   
 That sort of recipe blog hasn't happened just for SEO. It's also a bit of a "two audiences" problem: if you are coming to that food blogger from a search you certainly would prefer the recipe first and then maybe any commentary on it below if the recipe looks good. If you are a regular reader of that food blogger you are probably invested in the stories up top and that parasocial connection and the recipes themselves are sometimes incidental to why you are a regular reader.           
                   
 You see some of that "two readers" divide sometimes even in classic cookbooks, where "celebrity" chefs of the day might spend much of a cookbook on a long rambling memoir. Admittedly such books were generally well indexed and had table of contents to jump right to the recipes or particular recipes, but the concept of "long personal ramble of what these recipes mean to me" is an old one in cookbooks too.             
                   
                   
 inanutshellus on Sept 16, 2021 | root | parent | next [–]                  
                   
 I see your point, but argue you've misidentified the two audiences.                  
                   
 One audience matches your description and is the invested reader. They want that blogger's story telling. they might make the recipe, but they're a dedicated reader.                
                   
 The other audience is not the recipe-searcher, but instead Google. Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on. They won't even remember the blog's name. So the site isn't optimized for them. It's optimized for Google.               
                   
 "Slow the parasitic recipe-searcher down. They're leeches, here for a freebie. Well they'll pay me in Google Rank time blocks."                 
                   
                   
 hdjjhhvvhga on Sept 18, 2021 | root | parent | next [–]                  
                   
 > Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on.                 
                   
 This is not entirely true, though. If a randomly found recipe turns out particularly good, I'll bookmark the site and try out other dishes. It's a very practical method to find particularly good* recipe collections.                
                   
 *) In this case "good" means what you need - not just subjectively "tasty", but e.g. low cost, quick to prepare, low calorie or in line with a particular diet and so on.                
                   
                   
 inanutshellus on Sept 20, 2021 | root | parent | next [–]                  
                   
 I'm aware of zero human-behavior-truisms that are "entirely true".                  
                   
                   
 giantrobot on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 > If you are a regular reader of that food blogger                  
                   
 I think this assumes facts not in evidence. It certainly seems like an overwhelming number of "blogs" are not actual blogs but SEO content farms. There's no regular readers of such things because there's no actual authors, just someone that took a job on Fivver to spew out some SEO garbage. Old content gets reposted almost verbatim because new results better according to Google.             
                   
 The only reason these "blogs" exist is to show ads and hopefully get someone's e-mail (and implied consent) for a marke....newsletter.                 
                   
                   
 WorldMaker on Sept 16, 2021 | root | parent | next [–]                  
                   
 I know at least a few that I commonly see in top search results that I have friends that read them like personalized soap operas where most of the drama revolves around food and family and serving food to family.                
                   
 It's at least half the business models of Food Network shows: aspirational kitchens and the people that live in them and also sometimes here's their recipes. (The other half being competitions, obviously.) I've got friends that could deliver entire doctoral theses on the Bon Appetit Test Kitchen (and its many YouTube shows and blogs) and the huge soap operatic drama of 2020's events where the entire brand milkshake ducked itself; falling into people's hearts as "feel good" entertainment early in 2020/the pandemic and then exploding very dramatically with revelations and betrayals that Fall.          
                   
 Which isn't to say that there aren't garbage SEO farms out there in the food blogging space as well, but a lot of the big ones people commonly complain about seeing in google's results do have regular fans/audiences. (ETA: And many of the smaller blogs want to have regular fans/audiences. It's an active influencer/"content creator" space with relatively low barrier to entry that people love. Everyone's family loves food, it's a part of the human condition.)            
                   
                   
 run-types on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 I've basically never been taken to a recipe without a rambling preamble from Google. While food blogs may serve two audiences, a long introduction seems to be a requirement to appear in the top Google search results.                
                   
                   
 WorldMaker on Sept 16, 2021 | root | parent | next [–]                  
                   
 Personally, I think that has a lot more to do with the fact that Google killed the Recipe Databases. There did used to be a few startups that tried to be Recipe Aggregators with advertising based business models, that would show recipes and then link to source blogs and/or cookbooks, and in the brief period where they existed Google scraped them entirely and showed entire recipes on search results and ate their ad revenue out from under them.            
                   
                   
 dspillett on Sept 17, 2021 | root | parent | next [–]                  
                   
 Such databases would get battered by demands to remove content these days, if not already back then. No one want a database listing their stuff for ad revenue like that because many wouldn't follow the links so see their adverts or be subject to their tracking.               
                   
 A couple of browser add-ons specifically geared around trimming recipe pages down have been taken down due to similar complaints.                 
                   
                   
 tomrod on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 That is a really bad thing by Google. Their core business is not recipes.                  
                   
                   
 kwertyoowiyop on Sept 16, 2021 | root | parent | next [–]                  
                   
 Their core business is making money from other people’s content, no matter what it is.                  
                   
                   
 WorldMaker on Sept 16, 2021 | root | parent | next [–]                  
                   
 Their core business is advertising and they have always been in a direct conflict-of-interest by competing with content sites for ad revenue buys.                 
                   
                   
 xtracto on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 That's why I use Saffron [1], it magically converts those sites into a page in my recipe book. I found it when the developer commented here in HN. Also, a lot of cooking website have started to add a link with "jump to recipe" functionality allowing you to skip all the crap.               
                   
 [1] https://www.mysaffronapp.com/                  
                   
                   
 Funes- on Sept 16, 2021 | root | parent | next [–]                  
                   
 There's also https://based.cooking .                  
                   
                   
 ggggtez on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 I've noticed this pattern start to pop up elsewhere. I've started to train my skimming skills, skipping a paragraph or two at a time to get past the fluff.                
                   
 Like an article about some current event will undoubtedly begin with "when I was traveling ten years ago...".                 
                   
                   
 zerd on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 It's also because that's a way of trying to copyright protect recipes, which are normally not copyright protected.                 
                   
 > “Mere listings of ingredients as in recipes, formulas, compounds, or prescriptions are not subject to copyright protection. However, when a recipe or formula is accompanied by substantial literary expression in the form of an explanation or directions, or when there is a combination of recipes, as in a cookbook, there may be a basis for copyright protection.”             
                   
                   
 JohnFen on Sept 17, 2021 | root | parent | next [–]                  
                   
 But that copyright protection only extends to the literary expression. The recipe itself is still not covered by copyright, even if accompanied by an essay.                
                   
                   
 SerLava on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 That's not really for SEO, which favors readily accessible information.                  
                   
 That's ads. When mobile users have to scroll past 10 add, theyll click on some of them and make the blog money.                 
                   
                   
 Aeolun on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Searching for ‘chocolate’ on this search engine turned up a surprisingly large amount of chocolate based recipes.                 
                   
                   
 YeGoblynQueenne on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 >> This sort of optimization is why simple recipes are typically found at the end of a rambling pointless blog post now.                 
                   
 I continue to be curious about this kind of complaint. If all you want is a recipe list, without any of the fluff, why would you click on a link to a blog, rather than on a link to a recipe aggregator?                
                   
 Foodie blogs exist specifically for the people who want a foodie discussion and not just an ingredients' list.                 
                   
 Is it because blogs tend to have better recipes overall? In that case, isn't there a bit of entitlement involved in asking that the author self-sacrificingly provides only the information that you want, without taking care of their own needs and wants, also?               
                   
                   
 joegahona on Sept 16, 2021 | root | parent | next [–]                  
                   
 I think the complaint is that those blogs rank higher than nuts-and-bolts recipes now. It wasn't that way a few years ago. Yes, scrolling down the results to Food Network or Martha Stewart or whatever is possible, as is going directly to those sites and using their site search, but it's noticeable and annoying.              
                   
                   
 YeGoblynQueenne on Sept 17, 2021 | root | parent | next [–]                  
                   
 Not my experience. For a very quick test, I searched DDG for "omelette recipe, "carbonara recipe" and "peking duck recipe" (just to spice it up a bit) and all my top results are aggregators. Even "avgolemeono recipe" (which I'd think is very specialised) is aggregators on top.               
                   
 To be honest, I don't follow recipes when I cook unless it's a dish I've never had before. At that point what I want is to understand the point of the dish. A list of ingredients and preparation instructions don't tell me what it's supposed to taste and smell like. The foodie blogs at least try to create a certain... feeling of place, I suppose, some kind of impression that guides you when you cook. I wouldn't say it always works but I appreciate the effort.            
                   
 My real complaint with recipe writers is that they know how to cook one or two dishes well and they crib the rest off each other so even with all the information they provide, you still can't reliably cook a good meal from a recipe unless you've had the dish before. But that's my personal opinion.              
                   
                   
 jandrese on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Because when you search for a recipe you get the link to the blog, not the aggregator.                  
                   
                   
 Loughla on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 It's the same thing that people always complain about. This thing is not in a format that I like, so it must be not what anyone likes.                 
                   
 If you want JUST recipes, pay money instead of just randomly googling around. America's test kitchen has a billion, vetted, and really good recipes. That solves that problem.                
                   
                   
 marginalia_nu on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Specialization mostly a problem in monocultures.                  
                   
 If you almost only plant wheat, you are going to end up with one hell of a pest problem.                  
                   
 If you almost only have Windows XP, you are going to have one hell of a virus problem.                  
                   
 If you almost only have SearchRank-style search engines (or just the one), you are going to have one hell of a content spam problem.                 
                   
 Even though they have some pretty dodgy incentives, I don't think google suffers quality problems because they are evil, I think ultimately they suffer because they're so dominant. Whatever they do, the spammers adapt almost instantly.               
                   
 A diverse ecosystem on the other hand limits the viability of specialization by its very nature. If one actor is attacked, it shrinks and that reduces the opportunity for attacking it.                
                   
                   
 Nextgrid on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 I don't think the existing media-heavy websites are gaming Google to rank higher. It's that Google itself prefers media heavy content; they don't have to "game" anything.                
                   
 I also think a search engine like this would be quite hard to game. An ML-based classifier trained on thousands of text-heavy and media-heavy screenshots should be quite robust and I think would be very hard to evade, so the "game" will become more about how identify the crawler so you can serve it a high-ranking page while serving crap to the real users, and it seems fairly easily to defeat if the search engine does a second pass using residential proxies and standard browser user agents to detect this behavior (it could also threaten huge penalties like the entire domain being banned for a month to even deter attempts at this).         
                   
                   
 fragmede on Sept 16, 2021 | root | parent | next [–]                  
                   
 With the advances in text generation by machines that looks, but isn't quite accurate (aka GPT-3), seems like it would be easily gamed (given access to GPT-3). Even without GPT-3, if the content being prioritized is mere text, I'm sure that for a pile of money, I could generate something that looks like Wikipedia, in the sense that it's a giant pile of mostly text, but it would make zero sense to a human reader. (Building an SEO farm to boost ranking of not-wikpedia is left as an exercise for the reader.)           
                   
                   
 the_other on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 If there were a wider variety of popular search engines, with different ranking criteria, would sites begin to move away from gaming the system? Surely it would be too hard to game more than one search engine at a time?               
                   
                   
 Nasrudith on Sept 16, 2021 | root | parent | next [–]                  
                   
 It would be a matter of numbers anyway about which they optimize for. A/B testing is already in place and doesn't care about where it comes from, just which one does better.                
                   
                   
 phendrenad2 on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 There should be some perfect balance where this search engine is N% as popular as Google, where Google soaks up all of the gamifiers, but this search engine is still popular enough to derive revenue and do ML and other search-engine-useful stuff.               
                   
                   
 new_guy on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 > the people who want to game it will appear.                  
                   
 So just add human review to the mix, if a site is obviously trying to game the system (listicles, seo spam etc) just drop and ban them from the search index.                
                   
                   
 hdjjhhvvhga on Sept 16, 2021 | root | parent | next [–]                  
                   
 Congratulations, you've just invented negative SEO.                  
                   
                   
 rasz on Sept 17, 2021 | parent | prev | next [–]                  
                   
 >followed by a listicle "8 Reasons Why Rome Fell"                  
                   
 but arent you curious about the 7th reason? it will surprise you!                  
                   
                   
 coldtea on Sept 17, 2021 | root | parent | next [–]                  
                   
 You wont believe how Claudius looks today!                  
                   
                   
 kahrl on Sept 17, 2021 | root | parent | next [–]                  
                   
 Doctors HATE him!!!                   
                   
                   
 titzer on Sept 17, 2021 | parent | prev | next [–]                  
                   
 Search engines whose revenue is based on advertising will ultimately be tuned to steer you to the ad foodchain. All the incentives are aligned towards and all the metrics ultimately in service of, profit for advertisers. Not in the 99% of people who can convinced to consume something by ads? Welp, screw you.              
                   
                   
 phendrenad2 on Sept 17, 2021 | root | parent | next [–]                  
                   
 Search engines should be something you pay for. Surely search engine powerusers can afford to pay for such a service. If Google makes $1 per user per month or something, that's not too high a bar to get over.                
                   
                   
 titzer on Sept 17, 2021 | root | parent | next [–]                  
                   
 Search engines should be like libraries. At least some tiny sliver of the billions we spend on education and research should go to, you know, actually organizing the world's information and making it universally available.               
                   
                   
 illys on Sept 20, 2021 | root | parent | next [–]                  
                   
 I see another issue here: companies like Google prioritize information to 1) keep their users and 2) maximize their profit.                 
                   
 If you move data organization to another type of organization (non-profit, state, universities - private or public), then the question of data prioritization becomes highly political. What should be exposed? What should not? What to put first? ...               
                   
 It is already, but to a smaller extend since money-making companies have little interest in data meaning, and high interest in the commercial value of their users.                
                   
                   
 wizzwizz4 on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 In which case, consider paying for something like Infinity: https://infinitysearch.co/                  
                   
                   
 varjag on Sept 18, 2021 | root | parent | prev | next [–]                  
                   
 The theoretical cap for this, if you include every human being on planet Earth, is 7 billion/month. This translates into $84 billion annual revenue.                 
                   
 Google's revenue last year was 146 billion, and it operates not anywhere near the theoretical maximum. Most of that revenue is advertisement.                 
                   
                   
 Nition on Sept 16, 2021 | parent | prev | next [–]                  
                   
 The Wikipedia link at the top is always given. It would maybe be good to make it a little clearer that it's not one of the true results.                 
                   
                   
 Ajef on Sept 17, 2021 | root | parent | next [–]                  
                   
 I think this is just because of terms you have searched. In my test-searches Wikipedia has not come up once in first position (i think the highest was 3rd in the list).                
                   
 Here's what I've tried with a few variations: golang generics proposal, machine learning transformer, covid hospitalization germany                 
                   
 [edit] formatting                   
                   
                   
 Nition on Sept 17, 2021 | root | parent | next [–]                  
                   
 I think maybe it's a special insert at the top, but only if a Wikipedia page is found that matches you search term? I'm not sure now though.                 
                   
                   
 eterevsky on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Imagine if you were looking for the movie.                  
                   
                   
 MisterTea on Sept 16, 2021 | root | parent | next [–]                  
                   
 The you'd use a different search engine. Why does everything have to be a Swiss Army knife?                 
                   
                   
 zozbot234 on Sept 16, 2021 | root | parent | next [–]                  
                   
 Or you could just search for 'rome movie'. Though for more complex disambiguation you would need to resort to, e.g. schema.org descriptions (which are supported by most search engines, and the foundation for most "smart" search result snippets).               
                   
                   
 eterevsky on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 That's a fair point. This engine would be useful if you need grep over internet (by without regexes), i.e. when you want to find the exact phrases. But that's a relatively narrow use case.                
                   
                   
 lucideer on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 I tend to prefer Wikipedia for movies. The exception is actor headshots if I'm trying to identify someone, which Wikipedia lacks for licensing reasons, but otherwise Wikipedia tends to be better than IMDB for most needs. Wikipedia has an IMDB link on every article anyway.               
                   
 Another need I guess might be reviews, for which RT or MC are better than IMDB: not sure if either of those two will fare better than IMDB in this search engine but again Wiki has links out (in addition to good reception summaries)               
                   
                   
 mountainboy on Sept 16, 2021 | root | parent | next [–]                  
                   
 For me, imdb was much better when they had user comments/discussion.                  
                   
 I never even posted on it myself, but browsing the discussions one could learn all sorts of trivia, inside info, speculation, etc about each movie.                 
                   
 Since they (inexplicably) killed that feature, I rarely even visit anymore. Your right, for many purposes wikipedia is better, especially for TV series episode lists with summaries.                
                   
                   
 ncphil on Sept 16, 2021 | root | parent | next [–]                  
                   
 IMDB management thought it was their brilliant editorial work that drew people to their site. Morons. It was the comments all along. Of course they also believed they could create gravity-free zones by sheer force of executive will (and maybe still do).               
                   
                   
 _blu on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Especially for old and lesser known movies, the discussion board for the movie was a brilliant addition that could give the movie an extra dimension. Context is very important in order to understand, and ulitmately enjoy something.               
                   
 I think they removed it in part because new movies, like star wars and superhero movies, had alot of negative activity.                 
                   
                   
 sellyme on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 I find IMDb to be more convenient than RT/MC/Wikipedia for finding release dates of movies - nearly every other website lists only the American release date, maybe one or two others if the movie was disproportionately popular in certain regions.               
                   
                   
 neltnerb on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Imagine including the search term "movie".                  
                   
                   
 yreg on Sept 16, 2021 | root | parent | next [–]                  
                   
 That doesn't do anything useful.                  
                   
                   
 shuntress on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 ?q=imdb.com:fall of the roman empire                  
                   
                   
 jazzyjackson on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 !imdb                   
                   
                   
 purplefruit on Sept 17, 2021 | parent | prev | next [–]                  
                   
 Wow I used "personality test" and actually got useful articles about personality theory. I'll actually use this!                 
                   
                   
 HPsquared on Sept 18, 2021 | parent | prev | next [–]                  
                   
 I think it's a case where systems diversity can be an advantage. Much like how most malware was historically written for Windows and could be avoided by using Linux, the low-quality search engine bait is created for Google and can be avoided by using a different style of search engine.              
                   
                   
 psadri on Sept 17, 2021 | parent | prev | next [–]                  
                   
 Interesting choice of search topic. Are you trying to make an additional point?                  
                   
                   
 1vuio0pswjnm7 on Sept 19, 2021 | parent | prev | next [–]                  
                   
 No one mentioned the "bonus" audio in the page source: https://www.youtube.com/watch?v=7fCifJR6LAY                 
                   
                   
 marginalia_nu on Sept 19, 2021 | root | parent | next [–]                  
                   
 ;-)                   
                   
                   
 Siira on Sept 17, 2021 | parent | prev | next [–]                  
                   
 I tried some queries for Harry Potter fanfictions, and the results were pretty much completely unrelated. There weren’t that many results, either.                 
                   
                   
 marginalia_nu on Sept 18, 2021 | root | parent | next [–]                  
                   
 I'm curious what you searched for.                  
                   
 https://search.marginalia.nu/search?query=harry+potter+fanfi...                  
                   
 This seems to return a pretty decent number of sites relating to that (as well as some sites not relating to that).                 
                   
 The search engine isn't always great at knowing what a page is about, unfortunately.                  
                   
 This seemed to return mostly relevant results                  
                   
 https://search.marginalia.nu/search?query=%22harry+potter%22...                  
                   
                   
 Siira on Sept 19, 2021 | root | parent | next [–]                  
                   
 Yes, shorter queries return more relevant results. I think this was the first query that came to my mind:                 
                   
 https://search.marginalia.nu/search?query=Best+%22harry+pott...                  
                   
                   
 marginalia_nu on Sept 20, 2021 | root | parent | next [–]                  
                   
 Yeah, that's just not a type of query my search engine is particularly good at. It's pretty dumb, and just tries to match as much of the webpage against the query as it can.                
                   
 This used to be how all search engines worked, but I guess people have been taught by google that they should ask questions now, instead of search for terms.                
                   
 I wonder how I can guide people to make more suitable queries. Maybe I should just make it look less like google.                 
                   
                   
 hn_throwaway_99 on Sept 16, 2021 | parent | prev | next [–]                  
                   
 I had the exact opposite experience. I searched the site for "java", got a Wikipedia link first (for the island, not the programming language), and the 2nd result was to a random JEP page, and all the rest of the results were random tidbits about Java (e.g. "XZ compression algorithm in Java). Didn't get any high level results pointing to an overview of the language, getting started guides, etc.             
                   
                   
 withinboredom on Sept 16, 2021 | root | parent | next [–]                  
                   
 You need to use some old school search techniques and search for “Java overview”                  
                   
                   
 _wldu on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 I'm not sure that's a bad thing.                  
                   
                   
 rovr138 on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 well, they're results to java related items...                  
                   
 What kind of links where you expecting to find?                  
                   
                   
 acchow on Sept 17, 2021 | parent | prev | next [–]                  
                   
 If this search engine ever takes off, the listicle writers will just start optimizing for it too, right?                 
                   
                   
 dotancohen on Sept 17, 2021 | root | parent | next [–]                  
                   
 Mission accomplished, then.                  
                   
                   
 acchow on Sept 17, 2021 | root | parent | next [–]                  
                   
 If the goal was to remove modern web design, ok sure mission accomplished.                  
                   
 If your goal was to create a search engine that ignored listicles and other fluff and instead got you meatier results like "academic talks" and such, then no.                
                   
                   
 dotancohen on Sept 18, 2021 | root | parent | next [–]                  
                   
 When a measure becomes a target, it ceases to be a good measure.                  
                   
 https://en.wikipedia.org/wiki/Goodhart%27s_law                  
                   
                   
 adventured on Sept 16, 2021 | parent | prev | next [–]                  
                   
 I did a search for "George Washington"                  
                   
 First result after Wikipedia:                  
                   
 "Radiophone Transmitter on the U.S.S. George Washington (1920)                  
                   
 In 1906, Reginald Fessenden contracted with General Electric to build the first alternator transmitter. G.E. continued to perfect alternator transmitter design, and at the time of this report, the Navy was operating one of G.E.'s 200 kilowatt alternators http://earlyradiohistory.us/1919wsh.htm "              
                   
 Another result in the first few:                  
                   
 " - VANDERBILT, GEORGE WASHINGTON                  
                   
 PH: (800) ###-#233 FX: (#03) 641-5###. https://www.ScottWinslow.com/manufacturer/VANDERBILT_GEORGE_... "                 
                   
 And just below that terrible result:                  
                   
 "I Looked and I Listened -- George Washington Hill extract (1954)                  
                   
 Although the events described in this account are undated, they appear to have occurred in late 1928. I Looked and I Listened, Ben Gross, 1954, pages 104-105: Programs such as these called for the expenditure of larger sums than NBC had anticipated. It be http://earlyradiohistory.us/1954ayl2.htm "              
                   
 Dramatically worse than Google.                  
                   
 ---                   
                   
 Ok, how about a search for "Rome" then? Surely it'll pull some great text results for the city or the ancient empire.                 
                   
 First result after Wikipedia:                  
                   
 "Home | Rome Daily Sentinel                  
                   
 Reliable Community News for Oneida, Madison and Lewis County http://romesentinel.com/"                  
                   
 The fourth result for searching "Rome":                  
                   
 "Glenn's Pens - Stores of Note                  
                   
 Glenn's Pens, web site about pens, inks, stores, companies - the pleasure of owning and using a pen of choice. Direcdtory of pen stores in Europe. http://www.marcuslink.com/pens/storesofnote/roma.html"                
                   
 Again, dramatically worse than Google.                  
                   
 ---                   
                   
 Ok, how about if I search for "British"?                  
                   
 First result after Wikipedia:                  
                   
 "BRITISH MINING DATABASE                   
                   
 British_Mining_Database http://www.users.globalnet.co.uk/~lizcolin/bmd.htm "                  
                   
 And after that:                   
                   
 "British Virgin Islands                   
                   
 Many of these photos were taken on board the Spirit of Massachusetts. The sailing trip was organized by Toto Tours. Images Copyright © Lowell Greenberg Home Up Spring Quail Gardens Forest Home Lake Hodges Cape Falcon Cape Lookout, Oregon Wahkeena http://www.earthrenewal.org/british_virgin_islands2.htm"              
                   
 Again, far off the mark and dramatically worse than Google.                  
                   
 I like the idea of Google having lots of search competition, this isn't there yet (and I wouldn't expect it to be). I don't think overhyping its results does it any favors.                
                   
                   
 kwhitefoot on Sept 16, 2021 | root | parent | next [–]                  
                   
 What were you expecting to see for British? There must be millions of pages containing that term. Anyway the first screenful from Google is unadulterated crap, advertising mixed with the usual trivia questions.                
                   
 If you are going top claim something is wide of the mark then you really ought to tell us at least roughly where the mark is.                 
                   
                   
 burkaman on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 This is not a Google competitor, it's a different type of search engine with different goals.                 
                   
 > If you are looking for fact, this is almost certainly the wrong tool. If you are looking for serendipity, you're on the right track. When was the last time you just stumbled onto something interesting, by the way?                
                   
                   
 duckmysick on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 I checked the results of the same query and they seem fine. Lots of speeches and articles about George Washington the US president. There's even his beer recipe.                
                   
 As for the results you linked, it's part of the zeitgeist to list other entities sharing the same name. Sure, they could use some subtle changes in ranking, but overall the returned links satisfy my curiosity.                
                   
                   
 resynth1943 on Sept 17, 2021 | parent | prev | next [–]                  
                   
 Yeah, Google tends to send a lot of junk back.                  
                   
                   
 marginalia_nu on Sept 16, 2021 | prev | next [–]                  
                   
 Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)                
                   
                   
 crocodiletears on Sept 16, 2021 | parent | next [–]                  
                   
 It's very rare that I see a project on HN I can see myself using. This is one. Like others have said, the results can be a little rough. But they're rough in a way I think is much more manageable than the idiosynchrosies of more 'clever' search engines.               
                   
                   
 marginalia_nu on Sept 16, 2021 | root | parent | next [–]                  
                   
 I think you need to approach it more like grep than google. It's a forgotten art, dealing with this type of dumb search engine.                 
                   
 Like if you search for "How do I make a steak", you aren't going to get very good results. But a better query is "Steak Recipe", as that is at least a conceivable H1-tag.                
                   
                   
 crocodiletears on Sept 16, 2021 | root | parent | next [–]                  
                   
 This is exactly how I prefer to use my search engines.                  
                   
                   
 quaintdev on Sept 17, 2021 | root | parent | next [–]                  
                   
 I searched like this all my life and always got expected results.                  
                   
 But just a week ago I found out that these "how", "what" questions give better and faster results on Google.                 
                   
                   
 LeftHandPath on Sept 17, 2021 | root | parent | next [–]                  
                   
 That switch happened some years ago. I've been unlearning and relearning how to use google for what feels like at least three or four years now.                 
                   
 The main pain-point, though, is that a lot of long-tail searches you could've used to find different results in years past, now seem to funnel you to the same set of results based on your apparent intent. At least, it has felt that way -- I'm not entirely sure how the modern google algorithm works.              
                   
                   
 eitland on Sept 20, 2021 | root | parent | next [–]                  
                   
 I realized this a few years ago when I observed my wife find things faster on Google than me.                 
                   
 I appreciate that it is easier for newcomers but I still hate it personally after years and especially that they cannot even avoid meddling with my queries even when I try to accept the new system and use the verbatim option.               
                   
                   
 corobo on Sept 19, 2021 | root | parent | prev | next [–]                  
                   
 Try your old Google Fu skills on DuckDuckGo (or Bing I guess). I've found it to have good results anyway                 
                   
                   
 tomxor on Sept 19, 2021 | root | parent | prev | next [–]                  
                   
 > I think you need to approach it more like grep than google. It's a forgotten art                  
                   
 A search engine that accepted regex as the search parameter would be amazing.                  
                   
 I actually used this method as a field filter for a bunch of simple internal tools to search for info. Originally people were asking for individual search capabilities, but I didn't want it to become a giant project with me as the implementor of everyone's unique search capability feature request - so I just gave them regex, encoded inputs into the URL query string so they can save searches - gave em a bunch of examples to get going and now people are slowly learning regex and coming up with their own "new features" :P           
                   
 But this made sense because it's a relatively small amount of data, so small that it's searched in the front end which is why it's more of a filter... I don't think pure regex would scale when used as a query on a massive DB, it would need some kind of hierachy still to only bother parsing a subset of relevant text... unless there is some clever functional regex caching algorithm that can be used.             
                   
                   
 AQXt on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 So, you are re-implementing Altavista, Lycos and other old search engines.                  
                   
 They used the naive approach: you searched for "steak", and they would bring the pages which included the word "steak".                 
                   
 The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks.                
                   
 Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page.                 
                   
 The rest is history.                   
                   
                   
 frogpelt on Sept 17, 2021 | root | parent | next [–]                  
                   
 Effective Google search is also history.                  
                   
 I understand they are trying to maximize ad revenue and search does work very well for people who are looking for products or services.                 
                   
 But it no longer works well for finding information that is even slightly obscure.                  
                   
                   
 wolverine876 on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 > The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks.                
                   
 I don't see a lot of people investing in SEO to boost their Marginalia results.                  
                   
                   
 eitland on Sept 20, 2021 | root | parent | prev | next [–]                  
                   
 > Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page.                 
                   
 Then people fooled Google into showing the White House as top result when searching for "a miserable failure".                 
                   
 At the moment marginalia's approach of sorting pages into quality buckets based on lack of JS seems to be working extremely well, but of course it will be gamed if it gets popular.                
                   
 However, I'd rather want SEO-crafting to consider itself with minimizing JS, rather than spamming links into every comment field on every blog across the globe ;-)                
                   
                   
 asah on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Love it, kudos! This is great for developers and others who Just Need Answers and not shopping or entertainment.                 
                   
 If you're looking for feedback, both from a UI design and utility standpoint, you might consider "inlining" results from selected sites, e.g. Wikipedia, stacked change, etc. Having worked on search for a long time, inlining (onebox etc) is a big reason users choose Google, and that channelers fail to get traction. If you're Serious(tm), dog into the publisher structure formats and format those, create a test suite, etc.            
                   
 A word of caution: if this takes off, as a business it's vulnerable to Google shifting its algorithms slightly to identify the segment of users+queries who prefer these results and give the same results to those queries.               
                   
 Hope this helps!                   
                   
                   
 marginalia_nu on Sept 16, 2021 | root | parent | next [–]                  
                   
 If Google starts showing interesting text-heavy links instead of vapid listicles and storefronts, I have accomplished everything I ever could dream of.                 
                   
                   
 MaysonL on Sept 17, 2021 | root | parent | next [–]                  
                   
 Google Info - for when you're looking for information, not shopping advice or lists!                  
                   
                   
 wolverine876 on Sept 17, 2021 | root | parent | next [–]                  
                   
 Maybe you're joking, but this is a good idea for search engine. Better: Credible info.                  
                   
                   
 addandsubtract on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Google info? Can you give me a sample query of what you mean?                  
                   
                   
 kkiinnpptt on Sept 18, 2021 | root | parent | next [–]                  
                   
 It was a joke. The joke being that Google should launch at new product called Google Info, that would actually give you information when you search.                 
                   
                   
 palijer on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Haha, reminds me exactly of this.                  
                   
 https://xkcd.com/810/                   
                   
                   
 niutech on Sept 20, 2021 | root | parent | prev | next [–]                  
                   
 You can check the Web Vitals score of Google SERP-s using Core SERP Vitals ( https://chrome.google.com/webstore/detail/core-serp-vitals/o... ) and filter out the worst results.                
                   
                   
 0xbadcafebee on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 Thank you for doing this important work.                  
                   
                   
 aaron5 on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 haha, great answer! thanks for your work on this :)                  
                   
                   
 ad404b8a372f2b9 on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Very cool project! How many websites do you have in your index? And how did you go about building it?                 
                   
 I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.                
                   
                   
 marginalia_nu on Sept 19, 2021 | root | parent | next [–]                  
                   
 About 21 million, and I'm crawling myself.                  
                   
 Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text.                
                   
 I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.                
                   
                   
 ad404b8a372f2b9 on Sept 21, 2021 | root | parent | next [–]                  
                   
 If you're open to sharing your index I could make a classifier for you, I do this for a living. It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.               
                   
                   
 bityard on Sept 16, 2021 | parent | prev | next [–]                  
                   
 This is awesome. I've been looking for a long time for a search engine that basically takes everything Google does and does the opposite. Thank you for doing this, I will definitely be bookmarking it.                
                   
 Is there a way to suggest or add sites? I went looking for woodgears.ca and only got one result. I also think my personal blog would be a good candidate for being indexed here but I couldn't find any results for it.                
                   
                   
 Wronnay on Sept 19, 2021 | root | parent | next [–]                  
                   
 I would also love to add some sites which I were missing...                  
                   
 Unfortunately this doesn't seem to be a feature which new search engines are focusing on - Brave Search also misses that feature...                 
                   
                   
 edbaskerville on Sept 17, 2021 | parent | prev | next [–]                  
                   
 This has amazing potential. I'd encourage you to form a non-profit, turn this into something that can last as an organization without becoming what you're trying to avoid becoming. This is a good enough start that I bet you could raise a sizeable startup fund very soon from a combination of crowdfunding and foundation grants—I bet the Sloan Foundation would love this!             
                   
                   
 davegauer on Sept 16, 2021 | parent | prev | next [–]                  
                   
 This is absolutely wonderful. I am LOVING the results I'm getting back from it: the sort of content-rich sites that have become nigh unreachable using traditional search engines. Thank you for building this!                
                   
                   
 BugsJustFindMe on Sept 16, 2021 | parent | prev | next [–]                  
                   
 I love this, and I love (many of) the results so far! What I can't find on the site is detail about what "too many modern web design features" means. Is it just penalizing sites with tons of JavaScript?                
                   
                   
 marginalia_nu on Sept 16, 2021 | root | parent | next [–]                  
                   
 Javascript tags are penalized the hardest, but it also takes into consideration density of text per HTML. There's also some adjustments based on text length, which words occur in the page, etc.                
                   
                   
 santamex on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Which software do you use to index the sites?                  
                   
                   
 marginalia_nu on Sept 16, 2021 | root | parent | next [–]                  
                   
 I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke.                 
                   
 A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays.              
                   
 So when you search for two words, it translates them into IDs using the special hash map, goes to the words file and finds the least common of the words; starts with that.                
                   
 Then it goes to the words file and looks up the URL range of the first word.                  
                   
 Then it goes to the words file and looks up the URL range of the second word.                  
                   
 Then it goes through the less common word's range and does a binary search for each of those in the range of the more common word.                 
                   
 Then it grabs the first N results, and translates them into URLs (through mariadb); and that's your search result.                 
                   
 I'm skipping over a few steps, but that's the very crudest of outlines.                  
                   
                   
 q3k on Sept 16, 2021 | root | parent | next [–]                  
                   
 Good stuff. I've also been toying with doing some homegrown search engine indexing (as an exercise in scalable systems), and this is a fantastic result and great inspiration.                
                   
 Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.                
                   
                   
 marginalia_nu on Sept 16, 2021 | root | parent | next [–]                  
                   
 Well just crunching the numbers should indicate what is possible and what isn't.                  
                   
 For the moment I have just south of 20 million URLs indexed.                  
                   
 1 x 20 million bytes = 20 Mb.                  
                   
 10 x 20 million bytes = 200 Mb.                  
                   
 100 x 20 million bytes = 2 Gb.                  
                   
 1,000 x 20 million bytes = 20 Gb.                  
                   
 10,000 x 20 million bytes = 200 Gb.                  
                   
 100,000 x 20 million bytes = 2 Tb.                  
                   
 1,000,000 x 20 million bytes = 20 Tb.                  
                   
 This is still within what consumer hardware can deal with. It's getting expensive, but you don't need a datacenter to store 20 Tb worth of data.                 
                   
 How many bytes do you need, per document, for an index? Do you need 1 Mb of data to store index information about a page that, in terms of text alone, is perhaps 10 Kb?                
                   
                   
 freediver on Sept 18, 2021 | root | parent | next [–]                  
                   
 What crawler are you using and what kind of crawling speeds are you achieving?                  
                   
 How do you rank the results (is it based on content only) or you have external factors too?                 
                   
 What is your personal preferred search option of the 7 and why?                  
                   
 Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.                 
                   
                   
 marginalia_nu on Sept 19, 2021 | root | parent | next [–]                  
                   
 > What crawler are you using and what kind of crawling speeds are you achieving?                  
                   
 Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper.               
                   
 >How do you rank the results (is it based on content only) or you have external factors too?                 
                   
 I rank based on a pretty large number of factors, incoming links weighted by the "textiness" of the source domain, and similarity to the query.                 
                   
 > What is your personal preferred search option of the 7 and why?                  
                   
 I honestly use Google for a lot. My search engine isn't meant as a replacement, but a complement.                 
                   
 > Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.                 
                   
 Are you kidding? I think the Patreon is a resounding success! I'm still a bit stunned. I've gotten more support and praise, not just in terms of money but also emails and comments here than I could have ever dreamed possible.               
                   
 And this is just the start, too. I only recently got the search engine working this well. I have no doubt it can get much better. The fact that I have 11 people with me on that journey, even if they "just" pay my power bill, that's amazing.               
                   
 I'm honestly a bit at a loss for words.                  
                   
                   
 freediver on Sept 19, 2021 | root | parent | next [–]                  
                   
 You have a great attitude!                  
                   
 And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it?             
                   
 And what are the barriers to use something like this? You say yourself that you are using Google most of the time. Is jumping to check results on this engine going to be too much friction for most uses?                
                   
 Can something like this exist in isolation? What kind of value would it need to provide for users to remember using it en-masse as an additional/primary vertical search like they do for Amazon?                
                   
 Just thinking out-loud as I am also interested in the space (through http://teclis.com).                  
                   
                   
 marginalia_nu on Sept 19, 2021 | root | parent | next [–]                  
                   
 I think in part it may just be because I'm not trying to found a start-up, and I'm not trying to get rich quick. If I were, I would have dealt with this very differently. My integrity and the integrity of this project is far more important than my bank balance. Not everyone feels that way, and I can respect that, but I do.              
                   
 Ultimately I think running something like this for profit would create really unhealthy incentives to make my search engine worse. Any value it brings, right now, it brings because it isn't trying to cater to every taste and every use case.               
                   
 I also hate the constant "don't forget to slap the like and subscribe buttons"-shout outs of modern social media, even though I'm aware they it is extremely efficient. If I went down that route, I would become part of the problem I'm trying to help cure. I do feel the sirens' call though, it's intoxicating getting this sort of praise and attention.             
                   
 I want this to be a long-term project, not a some overnight cinderella story.                  
                   
 In the end, my search engine is never going to replace google. It isn't trying to, it's trying to complement it. It's barely able now, but hopefully I can make it much better in the months and years to come.                
                   
                   
 fuzzfactor on Sept 19, 2021 | root | parent | next [–]                  
                   
 I think it's good not to have to depend on financial compensation for every single thing in your life, if you can be comfortable or do well otherwise.                 
                   
 This allows quite a bit of its own kind of freedom even if maximum financial opportunity is not fully exploited. Perhaps even because you are not grasping for every dollar on the table at all times.                
                   
 You can do things without having to know if they will pay off, and if it turns out big anyway you can make money as a byproduct of what you do rather than having pure financial pursuit be the root of every goal.                
                   
                   
 freediver on Sept 19, 2021 | root | parent | prev | next [–]                  
                   
 I agree with everything you say. The 'subscribe and like buttons' would not help your conversion with HN readers, on the contrary. Trying to run this for profit also would not help your conversion with this audience.                
                   
 So given your setup is already ideal for 'conversions' for this population (low profile, high integrity, no BS) I was simply genuinely surprised that only 11 people converted given enormous visibility/interest this thread had. Hope that makes sense.               
                   
                   
 marginalia_nu on Sept 19, 2021 | root | parent | next [–]                  
                   
 I think it simply takes time to build trust. The threshold to sending someone money is high. I probably wouldn't send someone money based on a proof of concept and lofty ambitions alone.                
                   
 I'd absolutely consider sending someone money if they kept bringing something of value into my life. If I want more people to join the patreon, I'll just have to earn their trust and support.                
                   
                   
 fuzzfactor on Sept 19, 2021 | root | parent | prev | next [–]                  
                   
 The day Google first appeared on the full internet it was excellent of course because it had no ads.                 
                   
 Plus another excellent feature was you would get the same search results no matter who or where you were for quite some period of calendar time.                 
                   
 If something new did appear it was likely to be one of the new sites that was popping up all the time and it was likely to be as worthwhile as its established associates on the front page.                
                   
 You shouldn't need to crawl nearly as fast if you can compensate by treading more suitably where those have gone before.                 
                   
                   
 bullen on Sept 19, 2021 | root | parent | prev | next [–]                  
                   
 Interesting, in my database ( http://root.rupy.se ) I have one file per word that contains the ids (long) of the nodes (URLs), so to search many words together I have to go through the first file and one by one see if I find matches in the second.               
                   
 How does the range binary search work, does it just prune out the overlaps, how efficient is it and how much data do you have in there for say "hello" and "world" f.ex?                
                   
                   
 Aeolun on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 I’m not sure how you go from word to url range? Range implies contiguous, but how can you make that happen for a bunch of words without keeping track of a list of urls for each word (or URL ids, the idea is the same)?               
                   
                   
 marginalia_nu on Sept 17, 2021 | root | parent | next [–]                  
                   
 The trick is that the list of URLs for each word already is in the URLs file.                  
                   
 The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence.                
                   
 Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8.                 
                   
 The first range corresponds to the URLs 1, 5, 7; and the second 2, 5, 8.                  
                   
 If you search for hello world, it will first pick a range, the range for "hello", let's say (1,5,7); and then do binary searches in the second range -- the range corresponding to "world" -- (2,5,8) to find the overlap.               
                   
 This seems like it would be very slow, but since you can trivially find the size of the ranges, it's possible to always do them in an order of increasing range-sizes. 10 x log(100000) is a lot smaller than 100000 x log(10)               
                   
                   
 bullen on Sept 19, 2021 | root | parent | next [–]                  
                   
 Hm, ok I understand more but how do you perform the "binary search", just loop over the URL ids?                 
                   
 Funny I also selected "hello" and "world" above! Xo                  
                   
 My system is also written in Java btw!                  
                   
 Here are example results of my word search:                  
                   
 http://root.rupy.se/node/data/word/four                  
                   
 http://root.rupy.se/node/data/word/only                  
                   
 etc.                   
                   
                   
 marginalia_nu on Sept 19, 2021 | root | parent | next [–]                  
                   
 I'll get back to your email in a while, I've got a ton of messages I'm working through.                  
                   
 But yeah, in short pseudocode:                  
                   
 for url in range-for-"hello":                  
 if binary-search (range-for-"world", url):                  
 yield url                   
                   
 I do use streams, but that is the bare essence of it.                  
                   
                   
 soheil on Sept 21, 2021 | root | parent | prev | next [–]                  
                   
 So every time you insert a new URL for a word you have to update the range for every other single word since the URL file will be shifted?                 
                   
                   
 soheil on Sept 21, 2021 | root | parent | prev | next [–]                  
                   
 Are the n-grams always at most n=2 bigrams?                  
                   
                   
 marginalia_nu on Sept 21, 2021 | root | parent | next [–]                  
                   
 No, I actually count the n-grams as distinct words (up to 4-grams). The main limiter is for that is space, so I only extract "canned" n-grams from some tags.                
                   
 I would first search for the bigram hello_world, that's an O(1) array lookup; as then documents merely containing the words hello and world (usually not a good search result), that's the algorithm I'm describing in the parent comment.               
                   
                   
 soheil on Sept 21, 2021 | root | parent | next [–]                  
                   
 Makes sense. Every time you insert a new URL for a word you have to update the ranges for every other word since the URL file will be shifted?                 
                   
                   
 rvnx on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 It's a great project!                   
                   
                   
 agencies on Sept 18, 2021 | parent | prev | next [–]                  
                   
 I continue to want a communal set of 50-100 million urls and data that are "good" (i.e. for any value of good and more complete than common crawl) that are accessible enough to be easy to work with that can be used to experiment with different web search tech. There are enough separate problems to tackle in web search that breaking it down would maybe move the needle. We have lots of kaggle competitions about ranking, but using closed data. What other types of kaggle projects would help web search?           
                   
 What can we do to foster a sustainable bazaar of projects to make it easier to build web search engines?                 
                   
                   
 habibur on Sept 16, 2021 | parent | prev | next [–]                  
                   
 How are you doing the crawling without getting blocking? -- the hardest part.                  
                   
                   
 judge2020 on Sept 16, 2021 | root | parent | next [–]                  
                   
 Not OP but crawling is easy if you don't try scanning 5+ pages a second - almost all rate limiting/heuristic based 'keep server costs low' engines, including Cloudflare, don't care if you request every page, but will take action if you do something like burst every page and take up just as many server resources as a hundred concurrent users.              
                   
 Now, that is assuming you aren't on some VPS provider. If you're going to crawl, you'll have the best chance when you use your own IPs on your own ASN, with DNS and reverse DNS set up correctly. This makes it so the IP reputation systems can detect you as a crawler but not one that hammers every site it visits.              
                   
 Also, I imagine that, for a search engine like this, it doesn't expect content to change much anyways - so it can take its time crawling every site only once every month or two, instead of the multiple times a week (or day) search engines like Google have to for the constantly-updated content being churned out.              
                   
                   
 androceium on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Pretty neat!!!                   
                   
 You may already be aware of this, but the page doesn't seem to be formatted correctly on mobile. The content shows in a single thin column in the middle.                
                   
                   
 marginalia_nu on Sept 16, 2021 | root | parent | next [–]                  
                   
 Hmm, which OS? I only have a single Android phone so I've only fixed the CSS for that.                  
                   
                   
 androceium on Sept 16, 2021 | root | parent | next [–]                  
                   
 I was seeing it on Android w/ Firefox. Seems like it's fixed now though. :)                  
                   
                   
 marginalia_nu on Sept 17, 2021 | root | parent | next [–]                  
                   
 Curious, I haven't touched the stylesheets.                  
                   
                   
 ant6n on Sept 16, 2021 | root | parent | prev | next [–]                  
                   
 For example Firefox on Android.                  
                   
                   
 khimaros on Sept 17, 2021 | root | parent | prev | next [–]                  
                   
 Fennec F-Droid on Android 11 has some rendering issues.                  
                   
                   
 drpixie on Sept 18, 2021 | parent | prev | next [–]                  
                   
 Really like it, and love that you've done this yourself.                  
                   
 I'd prefer if it does just one thing and does that really well. Don't waste your time on calculator and conversion functions, or pseudo-natural language queries. There are plenty of good calculator & converter tools and websites, but we all need a really good search engine. I think you'd be better looking at handling synonyms and plurals.              
                   
 Thanks.                   
                   
                   
 marginalia_nu on Sept 18, 2021 | root | parent | next [–]                  
                   
 I've mostly added these converters and frills because I'm trying to eat my own dogfood, and google is deeply engrained as my go-to source for unit conversions and back-of-envelope calculations.                
                   
 Don't worry, this stuff is easy, and doesn't even remotely take away from the work on the harder problems.                 
                   
                   
 soheil on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Awesome project! How are you able to keep the site running after HN kiss of death? What is your stack, elastic search or something simper? How did you crawl so many websites for a project this size? Did you use any APIs like duck duck go or data from other search engines? Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?             
                   
                   
 marginalia_nu on Sept 17, 2021 | root | parent | next [–]                  
                   
 > How are you able to keep the site running after HN kiss of death?                  
                   
 I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware.                
                   
 My day job is also developing this sort of highly performance java applications, I guess it helps.                 
                   
 > What is your stack, elastic search or something simper?                  
                   
 It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries.                 
                   
 > How did you crawl so many websites for a project this size?                  
                   
 It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains).               
                   
 > Did you use any APIs like duck duck go or data from other search engines?                  
                   
 Nope, it's all me.                   
                   
 > Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?                 
                   
 I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains.               
                   
 It would be interesting to try a page rank-style approach, but my thinking is that because it's the algorithm, it's also the algorithm everyone is trying to game.                
                   
                   
 peterburkimsher on Sept 17, 2021 | parent | prev | next [–]                  
                   
 Thank you so much for creating such a useful search engine!                  
                   
 Is there any way that you can get an HTTP certificate?                  
                   
 I use an old iPhone 4S, and most of the modern web is inaccessible due to TLS. Hacker News and mbasic.facebook are two of the last sites I can use.                 
                   
 Usually text-based sites are more accessible, so this could be really useful to help me continue using my antique devices!                 
                   
                   
 agumonkey on Sept 16, 2021 | parent | prev | next [–]                  
                   
 is there a json endpoint ? I'd love to make an emacs bridge :)                  
                   
                   
 HellsMaddy on Sept 17, 2021 | root | parent | next [–]                  
                   
 Seconded, I’d like to incorporate it into a project of mine.                  
                   
                   
 zizee on Sept 17, 2021 | parent | prev | next [–]                  
                   
 Love the idea. A little feedback: layout needs tweaking for mobile. FWIW: I'm on mobile Firefox for Android.                 
                   
                   
 scrollaway on Sept 16, 2021 | parent | prev | next [–]                  
                   
 I searched Warcraft and got a gold selling/ level boosting site. Some things never change :)                 
                   
                   
 powerchord on Sept 18, 2021 | parent | prev | next [–]                  
                   
 This is really cool! A spacer between the links would help my old eyes; I keep getting lost in which link goes with which description. :-)                 
                   
                   
 axelroze on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Hi,                   
                   
 Interesting idea. Definitely see an overlap with eReader markets and looking at text only contents.                 
                   
 How does it work?                   
                   
 It ignores pages on which it detects frameworks for ui and ads or any javascript code at all?                 
                   
                   
 karlzt on Sept 18, 2021 | parent | prev | next [–]                  
                   
 I can't see the letters in the disturbing white search box, I'm on duckduckgo, brave, monocles, jquarks, smartcookieweb on Android.                 
                   
                   
 artembugara on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Nice, what are you using to crawl the web?                  
                   
                   
 marginalia_nu on Sept 16, 2021 | root | parent | next [–]                  
                   
 It's pretty much all bespoke.                  
                   
 I use external libraries for parsing HTML (JSoup) and robots.txt; but that's about it.                  
                   
                   
 soheil on Sept 17, 2021 | root | parent | next [–]                  
                   
 What was the starting site you fed to the crawler to follow the links from to build the index?                 
                   
                   
 marginalia_nu on Sept 17, 2021 | root | parent | next [–]                  
                   
 Just my (swedish) personal website. The first iteration of the search engine was probably mainly seeded by these links:                 
                   
 https://www.marginalia.nu/00-l%C3%A4nkar/                  
                   
 But I've since expanded my websites, so now I think these play a decent role in later iterations, although they are virtually all of them pages I've found eating my own dogfood:                
                   
 https://memex.marginalia.nu/links/fragments-old-web.gmi                  
                   
 https://memex.marginalia.nu/links/bookmarks.gmi                  
                   
                   
 niutech on Sept 18, 2021 | parent | prev | next [–]                  
                   
 Nice project, but have you heard of FrogFind? it also presents lightweight search results.                 
                   
                   
 bluefox on Sept 16, 2021 | parent | prev | next [–]                  
                   
 This is a very cool project! Thank you.                  
                   
                   
 aoms on Sept 18, 2021 | parent | prev | next [–]                  
                   
 What's the tech stack?                   
                   
                   
 marginalia_nu on Sept 18, 2021 | root | parent | next [–]                  
                   
 It's custom java code, for the most part. I'm using mariadb for some ancillary information, but the index and the crawler and everything is written from scratch.                
                   
 It's hosted on my consumer-equipment server (Ryzen 3900x, 128Gb ram, Optane 900p+a few IronWolf drives), bare bones on Debian.                 
                   
                   
 noduerme on Sept 16, 2021 | parent | prev | next [–]                  
                   
 I love this idea, and admire the work you put into it. I'm a fan of long reads and historical non-fiction, and Google's results are truly garbage.                 
                   
 I have a criticism that I think may pertain to the ranking methodology. I searched for "discovery of Australia". Among the top results were:                 
                   
 * A site claiming that the biblical flood was caused by Earth colliding with a comet (with several other pages from that site also making the top search results with other wild claims, e.g. that the Egyptians discovered Arizona);               
                   
 * Another site claiming the first inhabitants of Australia were a lost tribe of Israel;                  
                   
 * A third site claiming that Australia was discovered and founded by members of a secret society of Rosicrucians who had infiltrated the Dutch East India Company and planned to build an Australian utopia...                
                   
 These were all pages heavy with HTML4 tags and virtually devoid of Javascript, the kinds of pages you'd frequently see in the late 1990s from people who had built their own static websites in a text editor, or exported HTML from MS Word. At that time, there were millions of those sites with people paying for their own unique domain names, and so the proportion of them that were home to wild-eyed conspiracy theories was relatively small. What I think has happened is that kooks continued to keep these sites up - to the point where it's almost a visual trope now to see a red <h1> tag in Times New Roman and think, uh oh, I've stumbled on an "ancient aliens" site. Whereas scholars and journals offering higher quality information have moved to more modern platforms that rely more heavily on modern browsers - with or without their own domain names. So as a result what seemed to surface here were the fragments of the old web that remain live - possibly because people living in cabins in Montana forget to cancel their web hosting, or because the nature of old-school conspiracy theorists is to just keep packing their old sites with walls of text surrounded by <p> tags.
                   
 Arguably, this seems to rank the way Google's engine used to, since it couldn't run JS and they wanted to punish sites that used code to change markup at render time. At least, when I used to have to do onsite SEO work, it was always about simple tag hierarchies.               
                   
 I wonder whether there isn't some better metric of validity and information quality than what markup is used. Some of the sites that surfaced further down could be considered interesting and valuable resources. I think not punishing simple wall-of-text content is a good thing. But to punish more complicated layouts may have the perverse effect of downranking higher-quality sources of information - i.e. people and organizations who can afford to build a decent website, or who care to migrate to a modern blogging platform.           
                   
                   
 _dain_ on Sept 17, 2021 | root | parent | next [–]                  
                   
 those three pages sound pretty interesting, I don't see this as a problem                  
                   
                   
 GaylordTuring on Sept 18, 2021 | root | parent | prev | next [–]                  
                   
 I don’t want my search engine to somehow try to judge the believability of the results. I’d like to be the judge of that myself.                 
                   
                   
 ColinHayhurst on Sept 16, 2021 | parent | prev | next [–]                  
                   
 Great work. Working on an alternative search engine too. Take a look at my profile.                  
                   
                   
 blondin on Sept 16, 2021 | parent | prev | next [–]                  
                   
 fantastic project, thank you!                  
                   
                   
                   
 449 more comments...                   
                   
                   
                   
                   
                   
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact                  
                   
Search:                   

Wow, that's awesome. Great work!For a simple test, I searched "fall of the roman empire". In your search engine, I got wikipedia, followed by academic talks, chapters of books, and long-form blogs. All extremely useful resources. When I search on google, I get wikipedia, followed by a listicle "8 Reasons Why Rome Fell", then the imdb page for a movie by the same name, and then two Amazon book links, which are totally useless.
    
    
        
      
 Good comparison. Reminds me of an analogy I like to make of today's web, which is it feels like browsing through a magazine store — full of top 10s, shallow wow-factoids, and baity material. I genuinely believe terrible results like this are making society dumber.   
    
        
     The context matters. I'd happily read "Top 10" lists on a website if the site itself was dedicated to that one thing. "Top 10 Prog Rock albums", while a lazy, SEO-bait title, would at least be credible if it were on a music-oriented website.But no, these stories all come from cookie-cutter "new media" blog sites, written by an anonymous content writer who's repackaged Wikipedia/Discogs info into Buzzfeed-style copy writing designed to get people to "share to Twitter/FB". No passion, no expertise. Just eyeballs at any cost.
    
    
        
     This got me thinking that maybe one of the other big reasons for this is that the algorithms prioritize newer pages over older pages. This produces the problem where instead of covering a topic and refining it over time, the incentive is to repackage it over and over again.It reminds me of an annoyance I have with the Kindle store. If I wanted to find a book on, let's say, Psychology, there is no option to find all-time respected books of the past centenary. Amazon's algorithms constantly push to recommend the latest hot book of the year. But I don't want that. A year is not enough time to have society determine if the material withstands time. I want something that has stood the test of time and is recommended by reputable institutions.
    
    
        
     This is just a guess, but I believe that they use machine learning and rank it by the clicks. I took some coursera courses and Andrew Ng sort of suggested that as their strategy.The problem is that clickbait and low effort articles could be good enough to get the click, but low effort enough to drag society into the gutter. As time passes, the system is gamified more and more where the least effort for the most clicks is optimized.
    
    
    
        
      
 But they have. or could have. At least Google (and to a smaller extend Microsoft), if you are using Chrome/Bing have exactly that signal. If you stay on the site and scroll (taking time, reading, not skimming) all this could be a signal to evaluate if the search result met your needs.   
    
        
      
 I've heard google would guess with bounce rate. Or another way, if the user clicks on LinkedIn website A, after a few moments keeps trying other linksw/related search. It would mean it was not valuable.   
    
    
        
     &gt; is that the algorithms prioritize newer pages over older pages.They do? That would explain a lot - but ironically, I can't find a good source on this. Do you have one at hand?
    
    
        
      
 It is pretty obvious if you search for any old topic that is also covered incessantly by the news. "royal family" is a good example. There's no way those news stories published an hour ago are listed first due to a high PageRank score (which necessarily depends on time to accumulate inbound links).   
    
        
     It depends on the content. The flip side is looking up a programming-related question and getting results from 2012.I think they take different things into account based on the thing being searched.
    
    
        
      
 Even your example would depend upon the context. There are many cases where a programming question in 2021 is identical to one from 2012, along with the answer. In those instances, would you rather a shallow answer from 2021 or an indepth answer from 2012? This is not meant to imply that older answers offer greater depth, yet a heavy bias towards recent material can produce that outcome in some circumstances.   
    
        
      
 If you're using tools/languages that change rapidly (like Kotlin, in my case), syntax from a few years ago will often be outdated.   
    
        
      
 Yes, yet there are programming questions that go beyond "how do I do X in language Y" or "how do I do X with library Y". The language and library specific questions are the ones where I would be less inclined to want additional depth anyhow, well, provided they aren't dependent upon some language or library specific implementation detail.   
    
        
     There are of course a variety of factors, including the popularity of the site the page is published on. The signals related to the site are often as important as the content on the page itself. Even different parts of the same site can lend varying weight to something published in that section.Engagement, as measured in clicks and time spent on page, plays a big part. But you're right, to a degree, as frequently updated pages can rank higher in many areas. A newly published page has been recently updated. A lot depends on the (algorithmically perceived) topic too. Where news is concerned, you're completely right, algos are always going to favor newer content unless your search terms specify otherwise. PageRank, in it's original form, is long dead. Inbound link related signals are much more complex and contextual now, and other types of signals get more weight.
    
    
        
     Your Google search results show the date on articles do they not? If people are more likely to click on "Celebrity Net Worth (2021)" than "Celebrity Net Worth (2012)", then the algo will update to favour those results, because people are clicking on them.The only definitive source on this would be the gatekeeper itself. But Google never says anything explicitly, because they don't want people gaming search rankings. Even though it happens anyway.
    
    
        
     The new evergreen is refreshed sludge for bottom dollar. College kids stealing Reddit comments or moving around paragraphs from old articles. Or linking to linked blogs that link elsewhere.It's all stamped with Google Ads, of course, and then Google ranks these pages high enough to rake in eyeballs and ad dollars. Also there's the fact that each year, the average webpage picks up two more video elements / ad players, one or two more ad overlays, a cookie banner, and half a dozen banner/interstitials. It's 3-5% content spread thinly over an ad engine. The Google web is about squeezing ads down your throat.
    
    
        
      
 Really makes you wonder: you play whack a mole and tackle the symptoms with initiatives like this search engine. But the root of that problem and many many others is the same: advertising. Why don't we try to tackle that?   
    
    
    
    
        
     &gt; This got me thinking that maybe one of the other big reasons for this is that the algorithms prioritize newer pages over older pages.Actually that's not always the case. We publish a lot of blog content and it's really hard to publish new content that replaces old articles. We still see articles from 2017 coming up as more popular than newer, better treatments of the same subject. If somebody knows the SEO magic to get around this I'm all ears.
    
    
    
        
     Its the "healthy web" Mozilla^1 and Google keep telling their blog audiences about. :)1 Accept quid pro quo to send all queries to Google by default If what these companies were telling their readers was true, i.e., that advertising is "essential" for the web to survive, then how are the sites returned by this search engine for text-heavy websites (that are not discoverable through Google, the default search engine for Chrome, Firefox, etc.) able to remain online. Advertising is essential for the "tech" company middleman business to survive.
    
    
        
      
 I'm not sure I agree with your example. It seems to me it is the exact same as a "Top ten drinks to drink on a rainy day" list. There's simply too many good albums and opinions differ, so a top ten would -just like the drinks- end up being a list of the most popular ones with maybe one the author picks to stir some controversy or discussion. In my opinion the world would be a smarter place if Google ranked all such sites low. Then we might at least get fluff like "Top ten prog rock albums if you love X, hate Y and listen to Z when no one is around" instead.   
    
        
     Google won't rank them low because they actually do serve an important purpose. They're there for people who don't really know what they want specifically, they're looking for an overview. A top 10 gives a digestible overview on some topic, which helps the searcher narrow down what they really want.A "Top 10 albums of all time" post is actually better off going through 10 genres of popular music from the past 50 years and picking the top album (plus mentioning some other top albums in the genre) for each one. That gives the user the overview they're probably looking for, whether those are the top 10 albums of all time or not. It's a case of what the user searched for vs what they actually really want.
    
    
    
        
     So did Tim Berners-Lee. He was vehemently opposed to people shoehorning images into the WWW, because he didn't want it to turn into the equivalent of magazines. Which, I believe, he shared the opinion of them making society dumber.Appropriately enough, I couldn't find a good quote to verify that since Google is only giving me newspapers and magazines talking about Sir Tim in the context of current events. I do believe it's in his book "Weaving the Web" though.
    
    
        
     It's also possible that it's the other way around: a certain "common denominator" + algorithms that chase broad engagement = mediocre results.The real trick would be some kind of engine that can aim just above where the user's at.
    
    
        
     &gt; I genuinely believe terrible results like this are making society dumber.You have to e causality reversed. Google results reflect the fact that society is dumb.
    
    
    
    
        
      
 what I really want is a true AI to search through all that and figure out the useful truth. I don't know how to do this (and of course whoever writes the AI needs to be unbiased...)   
    
        
     &gt;whoever writes the AI needs to be unbiased...)I'm not sure the idea of a sentient being not having a bias is meaningful. Reality, once you get past the trivial bits, is subjective.
    
    
        
     Isn't there a fundamental ML postulate that learning without bias is impossible?Maybe not the same kind of bias we think of in terms of politics and such, but I wonder if there's a connection.
    
    
        
     I didn't say the AI should be unbiased, just whoever writes it.I want an AI that is biased to the truth when there is an objective one, and my tastes otherwise. (that is when asked to find a good book it should give me fantasy even though romance is the most popular genre and so will have better reviews)
    
    
        
      
 I think that is the goal, it's just what we currently have is an AI that's like a naive child who is easily tricked and distracted by clickbait.   
    
        
     &gt;an AI that's &lt;snip&gt; easily tricked and distracted by clickbait.So, AIs are actually on par with most adults now? (Sorry)
    
    
        
      
 Cool, it appears that the trend towards JS may be causing self-selection -- if a page has a high amount of JS, it is highly unlikely to contain anything of value.   
    
        
      
 True. Unfortunately many large corporate websites through which you pay bills, order tickets, etc. are becoming infested with JS widgets and bulky, slow interfaces. These are hard to avoid.   
    
        
      
 Conversely no software to install. Browser as a platform. Don’t have to boot to Windows to pay your bills with activex for example   
    
        
     The mostly JS-less web was fine, fast, and reliable 20 years ago and I never had ActiveX.I hear stories about Flash and ActiveX but I literally never needed these to shop or pay bills online. Payments also didn't require scripts from a dozen domains and four redirects..
    
    
    
        
      
 The platform isn't the problem. The problem is with the amount of code that does something other than letting you "pay bills, order tickets, etc.".   
    
        
      
 Huh. A weighted algorithm, somewhere between Google and the one linked, where you could subtract from sites by amount of JavaScript might be interesting.   
    
        
      
 If one could create an metric of ad to content ratio from the js used, I would guess that would be a nice differentiator too.   
    
        
      
 Browsers should be cherry picking the most compelling things that people accomplish with complex code and supporting them as a native feature.
Maybe the Browser Wars aren’t keeping up anymore.   
    
    
        
      
 However, when searching for "haskell type inference algorithm" I get completely useless results.   
    
        
     That query is too long apparently. But if you shorten to "haskell type inference", I think it delivers on its promise:&gt; If you are looking for fact, this is almost certainly the wrong tool. If you are looking for serendipity, you're on the right track. When was the last time you just stumbled onto something interesting, by the way?
    
    
        
     The search engine doesn't do any type of re-ordering or synonym stuff, it only tires to construct different N-grams from the search query.So if you for example compare "SDL tutorial" with "SDL tutorials". On google you'd get the same stuff, this search engine, for better or worse doesn't. This is a design decision, for now anyway, mostly because I'm incredibly annoyed when algorithms are second-guessing me. On the other hand, it does mean you sometimes have to try different searches to get relevant results.
    
    
    
        
     I’m not against a stemmer, actually, just against the aggressive concordances (?) that Google now employs, like when it shows me X in Banach spaces (the classical, textbook case) when I’m specifically searching for X in Fréchet spaces (the generalization I want to find 
but am not sure exists); of course Banach spaces and Fréchet spaces are almost exclusively encountered in the same context, but it doesn’t mean that one is a popular typo for the other! (The relative rarity of both of these in the corpus probably doesn’t help. The farcical case is BRST, or Becchi-Rouet-Stora-Tyutin, in physics, as it is literally a single key away from “best” and thus almost impossible to search for.)On the other hand, Google’s unawareness of (extensive and ubiquitous) Russian noun morphology is essentially what allowed Yandex to exist: both 2011 Yandex and 2021 Google are much more helpful for Russian than 2011 Google. I suspect (but have not checked) that the engine under discussion is utterly unusable for it. English (along with other Germanic and Romance languages to a lesser extent) is quite unusual in being meaningfully searchable without any understanding of morphology, globally speaking.
    
    
        
      
 I thought you could fix that by enclosing "BRST" in quotes, but apparently not. DuckDuckGo (which uses Google) returns a couple of results that do contain "BRST" in a medical context, but most results don't contain this string at all. What's going on?   
    
        
     I’m not certain what DDG actually uses (wasn’t it Bing?), but in my experience from the last couple of months it ignores quotes substantially more eagerly than Google does. For this particular term, a little bit of domain knowledge helps: even without quotes, brst becchi, brst formalism, brst quantization or perhaps bv brst will get you reasonable results. (I could swear Google corrected brst quantization to best quantization a year ago, but apparently not anymore.) Searching for stuff in the context of BRST is still somewhat unpleasant, though.I... don’t think anything particularly surprising is happening here, except for quotes being apparently ignored? I’ve had it explained to me that a rare word is essentially indistinguishable from a popular misspelling by NLP techniques as they currently exist, except by feeding the machine a massive dictionary (and perhaps not even then). BRST is a thing that you essentially can’t even define satisfactorily without at the very least four years of university-level physics (going by the conventional broad approach—the most direct possible road can of course be shorter if not necessarily more illuminating). “Best” is a very popular word both generally and in searches, and the R key is next to E on a Latin keyboard. If you are a perfect probabilistic reasoner with only these facts for context (and especially if you ignore case), I can very well believe that your best possible course of action is to assume a typo. How to permit overriding that decision (and indeed how to recognize you’ve actually made one worth worrying about without massive human input—e.g. Russian adjectives can have more than 20 distinct forms, can be made up on the spot by following productive word-formation processes, and you don’t want to learn all of the world’s languages!) is simply a very difficult problem for what is probably a marginal benefit in the grand scheme of things. I just dislike hitting these margins so much.
    
    
        
      
 It would not be a difficult problem if they allowed the " " operator to work as they claim it does, or revive the + operator.   
    
        
     In English, maybe; in Russian, I frequently find myself reaching for the nonexistent “morphology but not synonyms” operator (as the same noun phrase can take a different form depending on whether it is the subject or the object of a verb, or even on which verb it is the object of); even German should have the same problem AFAIU, if a bit milder. I don’t dare think about how speakers of agglunative languages (Finnish, Turkish, Malayalam) suffer.(DDG docs do say it supports +... and even +"...", but I can’t seem to get them to do what I want.)
    
    
        
      
 Ah, OK. I don’t know anything about Russian. This is a hard problem. I think the solution is something like what you suggest: more operators allowing different transformations. Even in English, I would like a "you may pluralize but nothing else" operator.   
    
        
     Well it’s not that alien, it (along with the other Eastern Slavic languages, Ukrainian and Belarusian) is mostly a run-of-the-mill European language (unlike Finnish, Estonian or Hungarian) except it didn’t lose the Indo-European noun case system like most but instead developed even more cases. That is, where English or French would differentiate the roles of different arguments of a verb by prepositions or implicitly by position, Russian (like German and Latin) has a special axis of noun forms called “case” which it uses for that (and also prepositions, which now require a certain case as well—a noun form can’t not have a case like it can’t not have a number).There are six of them (nominal [subject], genitive [belonging, part, absence, “of”], dative [indirect object, recipient, “to”], accusative [direct object], instrumental [device, means, “by”], prepositional [what the hell even is this]), so you have (cases) × (numbers) = 6 × 2 = 12 noun forms, and adjectives agree in number and gender with their noun, but (unlike Romance languages) plurals don’t have gender, so you have (cases) × (numbers and genders) = 6 × (3 + 1) = 24 adjective forms. None of this would be particularly problematic, except these forms work like French or Spanish verbs: they are synthetic (case, number and gender are all a single fused ending, not orthogonal ones) and highly convoluted with a lot of irregularities. And nouns and adjectives are usually more important for a web search than verbs.
    
    
    
    
        
     Well yeah, English is kind of weird, but Finnish isn’t a Germanic language at all? It’s not even Indo-European, so even Hindi is ostensibly closer to English than Finnish. I understand Standard German (along with Icelandic) is itself a bit atypical in that it hasn’t lost its cases when most other Germanic languages did.Re compounds, I expected they would be more or less easy to deal with by relatively dumb splitting, similar to greedy solutions to the “no spaces” problem of Chinese and Japanese, and your link seems to bear that out. But yeah, cheers to more language-specific stuff in your indexing. /s
    
    
    
    
        
      
 Oh this sounds like it could be a really cool idea! This way it could also be subtly teaching users that the engine doesn't do automatic synonyms translation so it's worth experimenting; also kinda like giving the synonyms feature while still keeping user in full control.   
    
    
    
        
      
 Since it does not use synonyms, it looks like it is unable to answer "how's that thing called"-queries.   
    
    
        
     Definitely; We could create a meta search engine that queries them all, in desktop application format.Let's name it after a famous old scientist, and maybe add the year to prove it's modern: Galileo 2021.
    
    
    
        
     &gt; You can't combine a few different ranked lists and expect to get results better than any of the original ranked lists.I am skeptical of this application of the theorem. Here is my proposal: Take the top 10 Google and Bing results. If the top result from Bing is in the top 10 from Google, display Google results. If the top result from Bing is not in the top 10 from Google, place it at the 10th position. You'd have an algorithm that ties with Google, say 98% of the time, beats it say, 1.2% of the time, and loses .8% of the time.
    
    
        
      
 Right. Arrow's theorem just says it's impossible to do it in all cases. It's still quite possible to get an improvement in a large proportion of cases, as you're proposing.   
    
        
     I've had jobs tuning up the relevance of search engines with methods like https://ccc.inaoep.mx/~villasen/bib/AN%20OVERVIEW%20OF%20EVA...  and the first conclusion is "something that you think will improve relevance probably won't"; the TREC conference went for about five years before making the first real discovery  https://en.wikipedia.org/wiki/Okapi_BM25  It's true that Arrow's Theorem doesn't strictly apply, but thinking about it makes it clear that the aggregation problem is ill-defined and tricky. (e.g. note also that a ranking function for full text search might have a range of 0-1 but is not a meaningful number, like a probability estimate that a document is relevant, but it just means that a result with a higher score is likely to be more relevant than one with a lower score.) Another way to think about it is that for any given feature architecture (say "bag of words") there is an (unknown) ideal ranking function. You might think that a real ranking function is the ideal ranking function plus an error and that averaging several ranking functions would keep the contribution of the ideal ranking function and the errors would average out, but actually the errors are correlated. In the case of BM25 for instance, it turns out you have to carefully tune between the biases of "long documents get more hits because they have more words in them" and "short documents rank higher because the document vectors are spiky like the query the vectors". Until BM25 there wasn't a function that could be tuned up properly and just averaging several bad functions doesn't solve the real problem.
    
    
        
      
 Arrows theorem simply doesn't apply here. We don't need our personalized search results to satisfy the majority.   
    
        
      
 But in both cases you face the problem of aggregating preferences of many into one. In one case you are combining personal preferences in the other case aggregating ‘preferences’ expressed by search engines.   
    
        
     But search engines aren't voting to maximize the chances that their preferred candidate shows up on top. The mixed ranker has no requirement to satisfy Arrows integrity constraints. It has to satisfy the end user, which is quite possible in theory.Conditions the mixed ranker doesn't have to satisfy 
"ranking while also meeting a specified set of criteria: unrestricted domain, non-dictatorship, Pareto efficiency, and independence of irrelevant alternatives"
    
    
        
     Sure, but the problem that conventional IR ranking functions are not meaningful other than by ordering leads you to the dismal world of political economy where you can't aggregate people's utility functions. (Thus you can't say anything about inequality, only about Pareto efficiency)Hypothetically you could treat these functions as meaningful but when you try you find that they aren't very meaningful. For instance IBM Watson aggregated multiple search sources by converting all the relevance scores to "the probability that this result is relevant". A conventional search engine will do horribly in that respect, you can fit a logit curve to make a probability estimator and you might get p=0.7 at the most and very rarely get that, in fact, you rarely get p&gt;0.5. If you are combining search results from search engines that use similar approaches you know those p's are not independent so you can't take a large numbers of p=0.7's and turn that into a higher p. If you are using search engines that use radically different matching strategies (say they return only p=0.99 results with low recall) the Watson approach works, but you need a big team to develop a long tail of matching strategies. If you had a good p-estimator for search you could do all sorts of things that normal search engines do poorly, such as "get an email when a p&gt;0.5 document is added to the collection." For now alerting features are either absent or useless and most people have no idea why.
    
    
        
     That's an invalid application of this theorem. (It doesn't necessarily hold)Suppose there's an unambiguous ranked preference by all people among a set (webpages, ranking). Suppose one search engine ranks correctly the top 5 results and incorrectly the next 5 results, while another ranks incorrectly the top 5 and correctly the next 5. What can happen is that some there may be no universally preferred search engine (likely). In practice, as another commenter noted, you can also have most users prefer more a certain combination of results (that's not difficult to imagine, for example by combining top independent results from different engines for example).
    
    
    
        
     Yup, but so far no one got it.There was such an app in the early 2000's, before Google went mainstream, and Altavista-like engines were not good: Copernic 2000. I guess I'm officially old now.
    
    
        
     For years I wanted to try Copernic Summarizer. It seemed like it actually worked. Then software that did summaries disappeared, maybe? And about 5 years ago bots on Reddit were doing summaries of news stories (and then links in comments).This is a pattern I see over and over again, some research group or academics show that something can be done (summaries that make sense and are true summaries, evolutionary algorithm FPGA programming, real time gaze prediction, etc) and there's a few published code repos and a bit of news, then 'poof' - no where to be seen for 15 years or more.
    
    
    
    
    
    
    
    
        
      
  a magic pop sound is faintly audible as a new side project is appended to several lists Excellent, thank you!   
    
    
        
      
 As long as few people use it, it will be great. Rest assured that the moment it becomes popular, the people who want to game it will appear.   
    
        
     This sort of optimization is why simple recipes are typically found at the end of a rambling pointless blog post now.Still, the best way to break SEO is to have actual competition in the search space. As long as SEO remains focused on Google there is an opportunity for these companies to thrive by evading SEO braindamage.
    
    
        
     That sort of recipe blog hasn't happened just for SEO. It's also a bit of a "two audiences" problem: if you are coming to that food blogger from a search you certainly would prefer the recipe first and then maybe any commentary on it below if the recipe looks good. If you are a regular reader of that food blogger you are probably invested in the stories up top and that parasocial connection and the recipes themselves are sometimes incidental to why you are a regular reader.You see some of that "two readers" divide sometimes even in classic cookbooks, where "celebrity" chefs of the day might spend much of a cookbook on a long rambling memoir. Admittedly such books were generally well indexed and had table of contents to jump right to the recipes or particular recipes, but the concept of "long personal ramble of what these recipes mean to me" is an old one in cookbooks too.
    
    
        
     I see your point, but argue you've misidentified the two audiences.One audience matches your description and is the invested reader. They want that blogger's story telling. they might make the recipe, but they're a dedicated reader. The other audience is not the recipe-searcher, but instead Google. Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on. They won't even remember the blog's name. So the site isn't optimized for them. It's optimized for Google. "Slow the parasitic recipe-searcher down. They're leeches, here for a freebie. Well they'll pay me in Google Rank time blocks."
    
    
        
     &gt; Food bloggers know that recipe-searchers are there to drop in, get an ingredient list, and move on.This is not entirely true, though. If a randomly found recipe turns out particularly good, I'll bookmark the site and try out other dishes. It's a very practical method to find particularly good* recipe collections. *) In this case "good" means what you need - not just subjectively "tasty", but e.g. low cost, quick to prepare, low calorie or in line with a particular diet and so on.
    
    
    
        
     &gt; If you are a regular reader of that food bloggerI think this assumes facts not in evidence. It certainly seems like an overwhelming number of "blogs" are not actual blogs but SEO content farms. There's no regular readers of such things because there's no actual authors, just someone that took a job on Fivver to spew out some SEO garbage. Old content gets reposted almost verbatim because new results better according to Google. The only reason these "blogs" exist is to show ads and hopefully get someone's e-mail (and implied consent) for a marke....newsletter.
    
    
        
     I know at least a few that I commonly see in top search results that I have friends that read them like personalized soap operas where most of the drama revolves around food and family and serving food to family.It's at least half the business models of Food Network shows: aspirational kitchens and the people that live in them and also sometimes here's their recipes. (The other half being competitions, obviously.) I've got friends that could deliver entire doctoral theses on the Bon Appetit Test Kitchen (and its many YouTube shows and blogs) and the huge soap operatic drama of 2020's events where the entire brand milkshake ducked itself; falling into people's hearts as "feel good" entertainment early in 2020/the pandemic and then exploding very dramatically with revelations and betrayals that Fall. Which isn't to say that there aren't garbage SEO farms out there in the food blogging space as well, but a lot of the big ones people commonly complain about seeing in google's results do have regular fans/audiences. (ETA: And many of the smaller blogs want to have regular fans/audiences. It's an active influencer/"content creator" space with relatively low barrier to entry that people love. Everyone's family loves food, it's a part of the human condition.)
    
    
        
      
 I've basically never been taken to a recipe without a rambling preamble from Google. While food blogs may serve two audiences, a long introduction seems to be a requirement to appear in the top Google search results.   
    
        
      
 Personally, I think that has a lot more to do with the fact that Google killed the Recipe Databases. There did used to be a few startups that tried to be Recipe Aggregators with advertising based business models, that would show recipes and then link to source blogs and/or cookbooks, and in the brief period where they existed Google scraped them entirely and showed entire recipes on search results and ate their ad revenue out from under them.   
    
        
     Such databases would get battered by demands to remove content these days, if not already back then. No one want a database listing their stuff for ad revenue like that because many wouldn't follow the links so see their adverts or be subject to their tracking.A couple of browser add-ons specifically geared around trimming recipe pages down have been taken down due to similar complaints.
    
    
    
    
        
      
 Their core business is advertising and they have always been in a direct conflict-of-interest by competing with content sites for ad revenue buys.   
    
        
     That's why I use Saffron [1], it magically converts those sites into a page in my recipe book. I found it when the developer commented here in HN. Also, a lot of cooking website have started to add a link with "jump to recipe" functionality allowing you to skip all the crap.[1] https://www.mysaffronapp.com/
    
    
    
        
     I've noticed this pattern start to pop up elsewhere. I've started to train my skimming skills, skipping a paragraph or two at a time to get past the fluff.Like an article about some current event will undoubtedly begin with "when I was traveling ten years ago...".
    
    
        
     It's also because that's a way of trying to copyright protect recipes, which are normally not copyright protected.&gt; “Mere listings of ingredients as in recipes, formulas, compounds, or prescriptions are not subject to copyright protection. However, when a recipe or formula is accompanied by substantial literary expression in the form of an explanation or directions, or when there is a combination of recipes, as in a cookbook, there may be a basis for copyright protection.”
    
    
        
      
 But that copyright protection only extends to the literary expression. The recipe itself is still not covered by copyright, even if accompanied by an essay.   
    
        
     That's not really for SEO, which favors readily accessible information.That's ads. When mobile users have to scroll past 10 add, theyll click on some of them and make the blog money.
    
    
        
      
 Searching for ‘chocolate’ on this search engine turned up a surprisingly large amount of chocolate based recipes.   
    
        
     &gt;&gt; This sort of optimization is why simple recipes are typically found at the end of a rambling pointless blog post now.I continue to be curious about this kind of complaint. If all you want is a recipe list, without any of the fluff, why would you click on a link to a blog, rather than on a link to a recipe aggregator? Foodie blogs exist specifically for the people who want a foodie discussion and not just an ingredients' list. Is it because blogs tend to have better recipes overall? In that case, isn't there a bit of entitlement involved in asking that the author self-sacrificingly provides only the information that you want, without taking care of their own needs and wants, also?
    
    
        
      
 I think the complaint is that those blogs rank higher than nuts-and-bolts recipes now. It wasn't that way a few years ago. Yes, scrolling down the results to Food Network or Martha Stewart or whatever is possible, as is going directly to those sites and using their site search, but it's noticeable and annoying.   
    
        
     Not my experience. For a very quick test, I searched DDG for "omelette recipe, "carbonara recipe" and "peking duck recipe" (just to spice it up a bit) and all my top results are aggregators. Even "avgolemeono recipe" (which I'd think is very specialised) is aggregators on top.To be honest, I don't follow recipes when I cook unless it's a dish I've never had before. At that point what I want is to understand the point of the dish. A list of ingredients and preparation instructions don't tell me what it's supposed to taste and smell like. The foodie blogs at least try to create a certain... feeling of place, I suppose, some kind of impression that guides you when you cook. I wouldn't say it always works but I appreciate the effort. My real complaint with recipe writers is that they know how to cook one or two dishes well and they crib the rest off each other so even with all the information they provide, you still can't reliably cook a good meal from a recipe unless you've had the dish before. But that's my personal opinion.
    
    
    
        
     It's the same thing that people always complain about. This thing is not in a format that I like, so it must be not what anyone likes.If you want JUST recipes, pay money instead of just randomly googling around. America's test kitchen has a billion, vetted, and really good recipes. That solves that problem.
    
    
        
     Specialization mostly a problem in monocultures.If you almost only plant wheat, you are going to end up with one hell of a pest problem. If you almost only have Windows XP, you are going to have one hell of a virus problem. If you almost only have SearchRank-style search engines (or just the one), you are going to have one hell of a content spam problem. Even though they have some pretty dodgy incentives, I don't think google suffers quality problems because they are evil, I think ultimately they suffer because they're so dominant. Whatever they do, the spammers adapt almost instantly. A diverse ecosystem on the other hand limits the viability of specialization by its very nature. If one actor is attacked, it shrinks and that reduces the opportunity for attacking it.
    
    
        
     I don't think the existing media-heavy websites are gaming Google to rank higher. It's that Google itself prefers media heavy content; they don't have to "game" anything.I also think a search engine like this would be quite hard to game. An ML-based classifier trained on thousands of text-heavy and media-heavy screenshots should be quite robust and I think would be very hard to evade, so the "game" will become more about how identify the crawler so you can serve it a high-ranking page while serving crap to the real users, and it seems fairly easily to defeat if the search engine does a second pass using residential proxies and standard browser user agents to detect this behavior (it could also threaten huge penalties like the entire domain being banned for a month to even deter attempts at this).
    
    
        
      
 With the advances in text generation by machines that looks, but isn't quite accurate (aka GPT-3), seems like it would be easily gamed (given access to GPT-3). Even without GPT-3, if the content being prioritized is mere text, I'm sure that for a pile of money, I could generate something that looks like Wikipedia, in the sense that it's a giant pile of mostly text, but it would make zero sense to a human reader. (Building an SEO farm to boost ranking of not-wikpedia is left as an exercise for the reader.)   
    
        
      
 If there were a wider variety of popular search engines, with different ranking criteria, would sites begin to move away from gaming the system? Surely it would be too hard to game more than one search engine at a time?   
    
        
      
 It would be a matter of numbers anyway about which they optimize for. A/B testing is already in place and doesn't care about where it comes from, just which one does better.   
    
        
      
 There should be some perfect balance where this search engine is N% as popular as Google, where Google soaks up all of the gamifiers, but this search engine is still popular enough to derive revenue and do ML and other search-engine-useful stuff.   
    
        
     &gt; the people who want to game it will appear.So just add human review to the mix, if a site is obviously trying to game the system (listicles, seo spam etc) just drop and ban them from the search index.
    
    
    
        
     &gt;followed by a listicle "8 Reasons Why Rome Fell"but arent you curious about the 7th reason? it will surprise you!
    
    
    
    
        
      
 Search engines whose revenue is based on advertising will ultimately be tuned to steer you to the ad foodchain. All the incentives are aligned towards and all the metrics ultimately in service of, profit for advertisers. Not in the 99% of people who can convinced to consume something by ads? Welp, screw you.   
    
        
      
 Search engines should be something you pay for. Surely search engine powerusers can afford to pay for such a service. If Google makes $1 per user per month or something, that's not too high a bar to get over.   
    
        
      
 Search engines should be like libraries. At least some tiny sliver of the billions we spend on education and research should go to, you know, actually organizing the world's information and making it universally available.   
    
        
     I see another issue here: companies like Google prioritize information to 1) keep their users and 2) maximize their profit.If you move data organization to another type of organization (non-profit, state, universities - private or public), then the question of data prioritization becomes highly political. What should be exposed? What should not? What to put first? ... It is already, but to a smaller extend since money-making companies have little interest in data meaning, and high interest in the commercial value of their users.
    
    
    
        
     The theoretical cap for this, if you include every human being on planet Earth, is 7 billion/month. This translates into $84 billion annual revenue.Google's revenue last year was 146 billion, and it operates not anywhere near the theoretical maximum. Most of that revenue is advertisement.
    
    
        
      
 The Wikipedia link at the top is always given. It would maybe be good to make it a little clearer that it's not one of the true results.   
    
        
     I think this is just because of terms you have searched. In my test-searches Wikipedia has not come up once in first position (i think the highest was 3rd in the list).Here's what I've tried with a few variations:
golang generics proposal,
machine learning transformer,
covid hospitalization germany [edit] formatting
    
    
        
      
 I think maybe it's a special insert at the top, but only if a Wikipedia page is found that matches you search term? I'm not sure now though.   
    
    
    
        
      
 Or you could just search for 'rome movie'. Though for more complex disambiguation you would need to resort to, e.g. schema.org descriptions (which are supported by most search engines, and the foundation for most "smart" search result snippets).   
    
        
      
 That's a fair point. This engine would be useful if you need grep over internet (by without regexes), i.e. when you want to find the exact phrases. But that's a relatively narrow use case.   
    
        
     I tend to prefer Wikipedia for movies. The exception is actor headshots if I'm trying to identify someone, which Wikipedia lacks for licensing reasons, but otherwise Wikipedia tends to be better than IMDB for most needs. Wikipedia has an IMDB link on every article anyway.Another need I guess might be reviews, for which RT or MC are better than IMDB: not sure if either of those two will fare better than IMDB in this search engine but again Wiki has links out (in addition to good reception summaries)
    
    
        
     For me, imdb was much better when they had user comments/discussion.I never even posted on it myself, but browsing the discussions one could learn all sorts of trivia, inside info, speculation, etc about each movie. Since they (inexplicably) killed that feature, I rarely even visit anymore. Your right, for many purposes wikipedia is better, especially for TV series episode lists with summaries.
    
    
        
      
 IMDB management thought it was their brilliant editorial work that drew people to their site. Morons. It was the comments all along. Of course they also believed they could create gravity-free zones by sheer force of executive will (and maybe still do).   
    
        
     Especially for old and lesser known movies, the discussion board for the movie was a brilliant addition that could give the movie an extra dimension. Context is very important in order to understand, and ulitmately enjoy something.I think they removed it in part because new movies, like star wars and superhero movies, had alot of negative activity.
    
    
        
      
 I find IMDb to be more convenient than RT/MC/Wikipedia for finding release dates of movies - nearly every other website lists only the American release date, maybe one or two others if the movie was disproportionately popular in certain regions.   
    
    
    
    
    
        
      
 Wow I used "personality test" and actually got useful articles about personality theory. I'll actually use this!   
    
        
      
 I think it's a case where systems diversity can be an advantage. Much like how most malware was historically written for Windows and could be avoided by using Linux, the low-quality search engine bait is created for Google and can be avoided by using a different style of search engine.   
    
    
    
    
        
      
 I tried some queries for Harry Potter fanfictions, and the results were pretty much completely unrelated. There weren’t that many results, either.   
    
    
    
        
     Yeah, that's just not a type of query my search engine is particularly good at. It's pretty dumb, and just tries to match as much of the webpage against the query as it can.This used to be how all search engines worked, but I guess people have been taught by google that they should ask questions now, instead of search for terms. I wonder how I can guide people to make more suitable queries. Maybe I should just make it look less like google.
    
    
        
      
 I had the exact opposite experience. I searched the site for "java", got a Wikipedia link first (for the island, not the programming language), and the 2nd result was to a random JEP page, and all the rest of the results were random tidbits about Java (e.g. "XZ compression algorithm in Java). Didn't get any high level results pointing to an overview of the language, getting started guides, etc.   
    
    
    
        
     well, they're results to java related items...What kind of links where you expecting to find?
    
    
        
      
 If this search engine ever takes off, the listicle writers will just start optimizing for it too, right?   
    
    
        
     If the goal was to remove modern web design, ok sure mission accomplished.If your goal was to create a search engine that ignored listicles and other fluff and instead got you meatier results like "academic talks" and such, then no.
    
    
    
        
     I did a search for "George Washington"First result after Wikipedia: "Radiophone Transmitter on the U.S.S. George Washington (1920) In 1906, Reginald Fessenden contracted with General Electric to build the first alternator transmitter. G.E. continued to perfect alternator transmitter design, and at the time of this report, the Navy was operating one of G.E.'s 200 kilowatt alternators
http://earlyradiohistory.us/1919wsh.htm
" Another result in the first few: " - VANDERBILT, GEORGE WASHINGTON PH: (800) ###-#233 FX: (#03) 641-5###.
https://www.ScottWinslow.com/manufacturer/VANDERBILT_GEORGE_...
" And just below that terrible result: "I Looked and I Listened -- George Washington Hill extract (1954) Although the events described in this account are undated, they appear to have occurred in late 1928. I Looked and I Listened, Ben Gross, 1954, pages 104-105: Programs such as these called for the expenditure of larger sums than NBC had anticipated. It be http://earlyradiohistory.us/1954ayl2.htm
" Dramatically worse than Google. --- Ok, how about a search for "Rome" then? Surely it'll pull some great text results for the city or the ancient empire. First result after Wikipedia: "Home | Rome Daily Sentinel Reliable Community News for Oneida, Madison and Lewis County
http://romesentinel.com/" The fourth result for searching "Rome": "Glenn's Pens - Stores of Note Glenn's Pens, web site about pens, inks, stores, companies - the pleasure of owning and using a pen of choice. Direcdtory of pen stores in Europe.
http://www.marcuslink.com/pens/storesofnote/roma.html" Again, dramatically worse than Google. --- Ok, how about if I search for "British"? First result after Wikipedia: "BRITISH MINING DATABASE British_Mining_Database
http://www.users.globalnet.co.uk/~lizcolin/bmd.htm
" And after that: "British Virgin Islands Many of these photos were taken on board the Spirit of Massachusetts. The sailing trip was organized by Toto Tours. Images Copyright © Lowell Greenberg Home Up Spring Quail Gardens Forest Home Lake Hodges Cape Falcon Cape Lookout, Oregon Wahkeena
http://www.earthrenewal.org/british_virgin_islands2.htm" Again, far off the mark and dramatically worse than Google. I like the idea of Google having lots of search competition, this isn't there yet (and I wouldn't expect it to be). I don't think overhyping its results does it any favors.
    
    
        
     What were you expecting to see for British? There must be millions of pages containing that term. Anyway the first screenful from Google is unadulterated crap, advertising mixed with the usual trivia questions.If you are going top claim something is wide of the mark then you really ought to tell us at least roughly where the mark is.
    
    
        
     This is not a Google competitor, it's a different type of search engine with different goals.&gt; If you are looking for fact, this is almost certainly the wrong tool. If you are looking for serendipity, you're on the right track. When was the last time you just stumbled onto something interesting, by the way?
    
    
        
     I checked the results of the same query and they seem fine. Lots of speeches and articles about George Washington the US president. There's even his beer recipe.As for the results you linked, it's part of the zeitgeist to list other entities sharing the same name. Sure, they could use some subtle changes in ranking, but overall the returned links satisfy my curiosity.
    
    
    
        
      
 Yeah so this is my project. It's very much a work in progress, but occasionally I think it works remarkably well for something I cobbled together alone out of consumer hardware and home-made code :-)   
    
        
      
 It's very rare that I see a project on HN I can see myself using. This is one. Like others have said, the results can be a little rough. But they're rough in a way I think is much more manageable than the idiosynchrosies of more 'clever' search engines.   
    
        
     I think you need to approach it more like grep than google. It's a forgotten art, dealing with this type of dumb search engine.Like if you search for "How do I make a steak", you aren't going to get very good results. But a better query is "Steak Recipe", as that is at least a conceivable H1-tag.
    
    
    
        
     I searched like this all my life and always got expected results.But just a week ago I found out that these "how", "what" questions give better and faster results on Google.
    
    
        
     That switch happened some years ago. I've been unlearning and relearning how to use google for what feels like at least three or four years now.The main pain-point, though, is that a lot of long-tail searches you could've used to find different results in years past, now seem to funnel you to the same set of results based on your apparent intent. At least, it has felt that way -- I'm not entirely sure how the modern google algorithm works.
    
    
        
     I realized this a few years ago when I observed my wife find things faster on Google than me.I appreciate that it is easier for newcomers but I still hate it personally after years and especially that they cannot even avoid meddling with my queries even when I try to accept the new system and use the verbatim option.
    
    
        
      
 Try your old Google Fu skills on DuckDuckGo (or Bing I guess). I've found it to have good results anyway   
    
        
     &gt; I think you need to approach it more like grep than google. It's a forgotten artA search engine that accepted regex as the search parameter would be amazing. I actually used this method as a field filter for a bunch of simple internal tools to search for info. Originally people were asking for individual search capabilities, but I didn't want it to become a giant project with me as the implementor of everyone's unique search capability feature request - so I just gave them regex, encoded inputs into the URL query string so they can save searches - gave em a bunch of examples to get going and now people are slowly learning regex and coming up with their own "new features" :P But this made sense because it's a relatively small amount of data, so small that it's searched in the front end which is why it's more of a filter... I don't think pure regex would scale when used as a query on a massive DB, it would need some kind of hierachy still to only bother parsing a subset of relevant text... unless there is some clever functional regex caching algorithm that can be used.
    
    
        
     So, you are re-implementing Altavista, Lycos and other old search engines.They used the naive approach: you searched for "steak", and they would bring the pages which included the word "steak". The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks. Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page. The rest is history.
    
    
        
     Effective Google search is also history.I understand they are trying to maximize ad revenue and search does work very well for people who are looking for products or services. But it no longer works well for finding information that is even slightly obscure.
    
    
        
     &gt; The problem is that people could fool these engines by adding a long sequence like "steak, steak, steak, steak, steak, steak" to their site -- to pretend that they were the most authoritative page about steaks.I don't see a lot of people investing in SEO to boost their Marginalia results.
    
    
        
     &gt; Google's big innovation was to count the referrers -- how many pages used the word "steak" to link to that particular page.Then people fooled Google into showing the White House as top result when searching for "a miserable failure". At the moment marginalia's approach of sorting pages into quality buckets based on lack of JS seems to be working extremely well, but of course it will be gamed if it gets popular. However, I'd rather want SEO-crafting to consider itself with minimizing JS, rather than spamming links into every comment field on every blog across the globe ;-)
    
    
        
     Love it, kudos! This is great for developers and others who Just Need Answers and not shopping or entertainment.If you're looking for feedback, both from a UI design and utility standpoint, you might consider "inlining" results from selected sites, e.g. Wikipedia, stacked change, etc. Having worked on search for a long time, inlining (onebox etc) is a big reason users choose Google, and that channelers fail to get traction. If you're Serious(tm), dog into the publisher structure formats and format those, create a test suite, etc. A word of caution: if this takes off, as a business it's vulnerable to Google shifting its algorithms slightly to identify the segment of users+queries who prefer these results and give the same results to those queries. Hope this helps!
    
    
        
      
 If Google starts showing interesting text-heavy links instead of vapid listicles and storefronts, I have accomplished everything I ever could dream of.   
    
    
    
    
        
      
 It was a joke. The joke being that Google should launch at new product called Google Info, that would actually give you information when you search.   
    
    
    
    
    
        
     Very cool project! How many websites do you have in your index? And how did you go about building it?I've been working on an engine for personal websites, currently trying to build a classifier to extract them from commoncrawl, if you have any general tips on that kind of project they'd be very welcome.
    
    
        
     About 21 million, and I'm crawling myself.Classification is really hard. I'm struggling with it myself, as a lot of like privacy policies and change logs turns out to share the shape of a page of text. I'm thinking of experimenting with ML classifiers, as I do have reasonably good ways of extracting custom datasets. Finding change logs and privacy policies is easy, excluding them is hard.
    
    
        
      
 If you're open to sharing your index I could make a classifier for you, I do this for a living. 
It's more of the indexing and search engine part which have been a problem for me. That's why I'm working from commoncrawls.   
    
        
     This is awesome. I've been looking for a long time for a search engine that basically takes everything Google does and does the opposite. Thank you for doing this, I will definitely be bookmarking it.Is there a way to suggest or add sites? I went looking for woodgears.ca and only got one result. I also think my personal blog would be a good candidate for being indexed here but I couldn't find any results for it.
    
    
        
     I would also love to add some sites which I were missing...Unfortunately this doesn't seem to be a feature which new search engines are focusing on - Brave Search also misses that feature...
    
    
        
      
 This has amazing potential. I'd encourage you to form a non-profit, turn this into something that can last as an organization without becoming what you're trying to avoid becoming. This is a good enough start that I bet you could raise a sizeable startup fund very soon from a combination of crowdfunding and foundation grants—I bet the Sloan Foundation would love this!   
    
        
      
 This is absolutely wonderful. I am LOVING the results I'm getting back from it: the sort of content-rich sites that have become nigh unreachable using traditional search engines. Thank you for building this!   
    
        
      
 I love this, and I love (many of) the results so far! What I can't find on the site is detail about what "too many modern web design features" means. Is it just penalizing sites with tons of JavaScript?   
    
        
      
 Javascript tags are penalized the hardest, but it also takes into consideration density of text per HTML. There's also some adjustments based on text length, which words occur in the page, etc.   
    
    
        
     I wrote it myself from scratch. I have some metadata in mariadb, but the index is bespoke.A design sketch of the index is that it uses one file with sorted URL IDs, one with IDs of N-grams (i.e. words and word-pairs) referring to ranges in the URL file; as well as a dictionary for relating words to word-IDs; that's a GNU Trove hash map I modified to use memory map data instead of direct allocated arrays. So when you search for two words, it translates them into IDs using the special hash map, goes to the words file and finds the least common of the words; starts with that. Then it goes to the words file and looks up the URL range of the first word. Then it goes to the words file and looks up the URL range of the second word. Then it goes through the less common word's range and does a binary search for each of those in the range of the more common word. Then it grabs the first N results, and translates them into URLs (through mariadb); and that's your search result. I'm skipping over a few steps, but that's the very crudest of outlines.
    
    
        
     Good stuff. I've also been toying with doing some homegrown search engine indexing (as an exercise in scalable systems), and this is a fantastic result and great inspiration.Definitely want to see more people doing that kind of low-level work instead of falling back to either 'use elasticsearch' or 'you can't, you're not google'.
    
    
        
     Well just crunching the numbers should indicate what is possible and what isn't.For the moment I have just south of 20 million URLs indexed. 1 x 20 million bytes = 20 Mb. 10 x 20 million bytes = 200 Mb. 100 x 20 million bytes = 2 Gb. 1,000 x 20 million bytes = 20 Gb. 10,000 x 20 million bytes = 200 Gb. 100,000 x 20 million bytes = 2 Tb. 1,000,000 x 20 million bytes = 20 Tb. This is still within what consumer hardware can deal with. It's getting expensive, but you don't need a datacenter to store 20 Tb worth of data. How many bytes do you need, per document, for an index? Do you need 1 Mb of data to store index information about a page that, in terms of text alone, is perhaps 10 Kb?
    
    
        
     What crawler are you using and what kind of crawling speeds are you achieving?How do you rank the results (is it based on content only) or you have external factors too? What is your personal preferred search option of the 7 and why? Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way.
    
    
        
     &gt; What crawler are you using and what kind of crawling speeds are you achieving?Custom crawler, and I seem to get around 100 documents per second at best, maybe closer to 50 on average. Depends a bit on how many crawl-worthy websites it finds, and there is definitely diminishing returns as it goes deeper. &gt;How do you rank the results (is it based on content only) or you have external factors too? I rank based on a pretty large number of factors, incoming links weighted by the "textiness" of the source domain, and similarity to the query. &gt; What is your personal preferred search option of the 7 and why? I honestly use Google for a lot. My search engine isn't meant as a replacement, but a complement. &gt; Thanks for making something unique and sorry that despite all the hype this got, you got only $39/month on Patreon. It is telling in a way. Are you kidding? I think the Patreon is a resounding success! I'm still a bit stunned. I've gotten more support and praise, not just in terms of money but also emails and comments here than I could have ever dreamed possible. And this is just the start, too. I only recently got the search engine working this well. I have no doubt it can get much better. The fact that I have 11 people with me on that journey, even if they "just" pay my power bill, that's amazing. I'm honestly a bit at a loss for words.
    
    
        
     You have a great attitude!And I am not kidding. I think for something that got so much attention on HN, where realistically this kind of product can only exist for now, the 'conversion' rate was very low. Billion dollar companies were made of HN threads with lot less engagement. Makes me wonder do we really want a search engine like this or we just like the idea of it? And what are the barriers to use something like this? You say yourself that you are using Google most of the time. Is jumping to check results on this engine going to be too much friction for most uses? Can something like this exist in isolation? What kind of value would it need to provide for users to remember using it en-masse as an additional/primary vertical search like they do for Amazon? Just thinking out-loud as I am also interested in the space (through http://teclis.com).
    
    
        
     I think in part it may just be because I'm not trying to found a start-up, and I'm not trying to get rich quick. If I were, I would have dealt with this very differently. My integrity and the integrity of this project is far more important than my bank balance. Not everyone feels that way, and I can respect that, but I do.Ultimately I think running something like this for profit would create really unhealthy incentives to make my search engine worse. Any value it brings, right now, it brings because it isn't trying to cater to every taste and every use case. I also hate the constant "don't forget to slap the like and subscribe buttons"-shout outs of modern social media, even though I'm aware they it is extremely efficient. If I went down that route, I would become part of the problem I'm trying to help cure. I do feel the sirens' call though, it's intoxicating getting this sort of praise and attention. I want this to be a long-term project, not a some overnight cinderella story. In the end, my search engine is never going to replace google. It isn't trying to, it's trying to complement it. It's barely able now, but hopefully I can make it much better in the months and years to come.
    
    
        
     I think it's good not to have to depend on financial compensation for every single thing in your life, if you can be comfortable or do well otherwise.This allows quite a bit of its own kind of freedom even if maximum financial opportunity is not fully exploited. Perhaps even because you are not grasping for every dollar on the table at all times. You can do things without having to know if they will pay off, and if it turns out big anyway you can make money as a byproduct of what you do rather than having pure financial pursuit be the root of every goal.
    
    
        
     I agree with everything you say. The 'subscribe and like buttons' would not help your conversion with HN readers, on the contrary. Trying to run this for profit also would not help your conversion with this audience.So given your setup is already ideal for 'conversions' for this population (low profile, high integrity, no BS) I was simply genuinely surprised that only 11 people converted given enormous visibility/interest this thread had. Hope that makes sense.
    
    
        
     I think it simply takes time to build trust. The threshold to sending someone money is high. I probably wouldn't send someone money based on a proof of concept and lofty ambitions alone.I'd absolutely consider sending someone money if they kept bringing something of value into my life. If I want more people to join the patreon, I'll just have to earn their trust and support.
    
    
        
     The day Google first appeared on the full internet it was excellent of course because it had no ads.Plus another excellent feature was you would get the same search results no matter who or where you were for quite some period of calendar time. If something new did appear it was likely to be one of the new sites that was popping up all the time and it was likely to be as worthwhile as its established associates on the front page. You shouldn't need to crawl nearly as fast if you can compensate by treading more suitably where those have gone before.
    
    
        
     Interesting, in my database (http://root.rupy.se) I have one file per word that contains the ids (long) of the nodes (URLs), so to search many words together I have to go through the first file and one by one see if I find matches in the second.How does the range binary search work, does it just prune out the overlaps, how efficient is it and how much data do you have in there for say "hello" and "world" f.ex?
    
    
        
      
 I’m not sure how you go from word to url range? Range implies contiguous, but how can you make that happen for a bunch of words without keeping track of a list of urls for each word (or URL ids, the idea is the same)?   
    
        
     The trick is that the list of URLs for each word already is in the URLs file.The URLs in a range are sorted. A sorted list (or list-range) forms an implicit set-like data structure, where you can do binary searches to test for existence. Consider a words file with two words, "hello" and "world", corresponding to the ranges (0,3), (3,6). The URLs file contains URLs 1, 5, 7, 2, 5, 8. The first range corresponds to the URLs 1, 5, 7; and the second 2, 5, 8. If you search for hello world, it will first pick a range, the range for "hello", let's say (1,5,7); and then do binary searches in the second range -- the range corresponding to "world" -- (2,5,8) to find the overlap. This seems like it would be very slow, but since you can trivially find the size of the ranges, it's possible to always do them in an order of increasing range-sizes. 10 x log(100000) is a lot smaller than 100000 x log(10)
    
    
    
        
      
 I'll get back to your email in a while, I've got a ton of messages I'm working through.But yeah, in short pseudocode:   for url in range-for-"hello":
 if binary-search (range-for-"world", url):
 yield url
 
I do use streams, but that is the bare essence of it.   
    
        
      
 So every time you insert a new URL for a word you have to update the range for every other single word since the URL file will be shifted?   
    
    
        
     No, I actually count the n-grams as distinct words (up to 4-grams). The main limiter is for that is space, so I only extract "canned" n-grams from some tags.I would first search for the bigram hello_world, that's an O(1) array lookup; as then documents merely containing the words hello and world (usually not a good search result), that's the algorithm I'm describing in the parent comment.
    
    
        
      
 Makes sense. Every time you insert a new URL for a word you have to update the ranges for every other word since the URL file will be shifted?   
    
    
        
     I continue to want a communal set of 50-100 million urls and data that are "good" (i.e. for any value of good and more complete than common crawl) that are accessible enough to be easy to work with that can be used to experiment with different web search tech. There are enough separate problems to tackle in web search that breaking it down would maybe move the needle. We have lots of kaggle competitions about ranking, but using closed data. What other types of kaggle projects would help web search?What can we do to foster a sustainable bazaar of projects to make it easier to build web search engines?
    
    
    
        
     Not OP but crawling is easy if you don't try scanning 5+ pages a second - almost all rate limiting/heuristic based 'keep server costs low' engines, including Cloudflare, don't care if you request every page, but will take action if you do something like burst every page and take up just as many server resources as a hundred concurrent users.Now, that is assuming you aren't on some VPS provider. If you're going to crawl, you'll have the best chance when you use your own IPs on your own ASN, with DNS and reverse DNS set up correctly. This makes it so the IP reputation systems can detect you as a crawler but not one that hammers every site it visits. Also, I imagine that, for a search engine like this, it doesn't expect content to change much anyways - so it can take its time crawling every site only once every month or two, instead of the multiple times a week (or day) search engines like Google have to for the constantly-updated content being churned out.
    
    
        
     Pretty neat!!!You may already be aware of this, but the page doesn't seem to be formatted correctly on mobile. The content shows in a single thin column in the middle.
    
    
    
    
    
    
    
        
     Really like it, and love that you've done this yourself.I'd prefer if it does just one thing and does that really well. Don't waste your time on calculator and conversion functions, or pseudo-natural language queries. There are plenty of good calculator &amp; converter tools and websites, but we all need a really good search engine. I think you'd be better looking at handling synonyms and plurals. Thanks.
    
    
        
     I've mostly added these converters and frills because I'm trying to eat my own dogfood, and google is deeply engrained as my go-to source for unit conversions and back-of-envelope calculations.Don't worry, this stuff is easy, and doesn't even remotely take away from the work on the harder problems.
    
    
        
      
 Awesome project! How are you able to keep the site running after HN kiss of death? What is your stack, elastic search or something simper? How did you crawl so many websites for a project this size? Did you use any APIs like duck duck go or data from other search engines? Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor?   
    
        
     &gt; How are you able to keep the site running after HN kiss of death?I originally targeted a Raspberry Pi4-cluster. It was only able to deal with about 200k pages at that stage, but it did shape the design in a way that makes very thrifty use of the available hardware. My day job is also developing this sort of highly performance java applications, I guess it helps. &gt; What is your stack, elastic search or something simper? It's a custom index engine I built for this. I do use mariadb for some ancillary data and to support the crawler, but it's only doing trivial queries. &gt; How did you crawl so many websites for a project this size? It's not that hard. Like it seems like it would be, and there certainly is an insane number of edge cases, but if you just keep tinkering you can easily crawl dozens of pages per second even on modest hardware (of course distributed across different domains). &gt; Did you use any APIs like duck duck go or data from other search engines? Nope, it's all me. &gt; Are you still incorporating something like PageRank to ensure good results are prioritized or is it just the text-based-ness factor? I'm using a somewhat convoluted algorithm that takes into consideration the text-based-ness of the page, but also how many incoming links the domain has, but it's a weighted value that factors in the text-based-ness of the origin domains. It would be interesting to try a page rank-style approach, but my thinking is that because it's the algorithm, it's also the algorithm everyone is trying to game.
    
    
        
     Thank you so much for creating such a useful search engine!Is there any way that you can get an HTTP certificate? I use an old iPhone 4S, and most of the modern web is inaccessible due to TLS. Hacker News and mbasic.facebook are two of the last sites I can use. Usually text-based sites are more accessible, so this could be really useful to help me continue using my antique devices!
    
    
    
    
        
      
 Love the idea. A little feedback: layout needs tweaking for mobile. FWIW: I'm on mobile Firefox for Android.   
    
    
        
      
 This is really cool! A spacer between the links would help my old eyes; I keep getting lost in which link goes with which description. :-)   
    
        
     Hi,Interesting idea. Definitely see an overlap with eReader markets and looking at text only contents. How does it work? It ignores pages on which it detects frameworks for ui and ads or any javascript code at all?
    
    
        
      
 I can't see the letters in the disturbing white search box, I'm on duckduckgo, brave, monocles, jquarks, smartcookieweb on Android.   
    
    
    
        
      
 What was the starting site you fed to the crawler to follow the links from to build the index?   
    
    
    
    
    
        
     It's custom java code, for the most part. I'm using mariadb for some ancillary information, but the index and the crawler and everything is written from scratch.It's hosted on my consumer-equipment server (Ryzen 3900x, 128Gb ram, Optane 900p+a few IronWolf drives), bare bones on Debian.
    
    
        
     I love this idea, and admire the work you put into it. I'm a fan of long reads and historical non-fiction, and Google's results are truly garbage.I have a criticism that I think may pertain to the ranking methodology. I searched for "discovery of Australia". Among the top results were: * A site claiming that the biblical flood was caused by Earth colliding with a comet (with several other pages from that site also making the top search results with other wild claims, e.g. that the Egyptians discovered Arizona); * Another site claiming the first inhabitants of Australia were a lost tribe of Israel; * A third site claiming that Australia was discovered and founded by members of a secret society of Rosicrucians who had infiltrated the Dutch East India Company and planned to build an Australian utopia... These were all pages heavy with HTML4 tags and virtually devoid of Javascript, the kinds of pages you'd frequently see in the late 1990s from people who had built their own static websites in a text editor, or exported HTML from MS Word. At that time, there were millions of those sites with people paying for their own unique domain names, and so the proportion of them that were home to wild-eyed conspiracy theories was relatively small. What I think has happened is that kooks continued to keep these sites up - to the point where it's almost a visual trope now to see a red &lt;h1&gt; tag in Times New Roman and think, uh oh, I've stumbled on an "ancient aliens" site. Whereas scholars and journals offering higher quality information have moved to more modern platforms that rely more heavily on modern browsers - with or without their own domain names. So as a result what seemed to surface here were the fragments of the old web that remain live - possibly because people living in cabins in Montana forget to cancel their web hosting, or because the nature of old-school conspiracy theorists is to just keep packing their old sites with walls of text surrounded by &lt;p&gt; tags. Arguably, this seems to rank the way Google's engine used to, since it couldn't run JS and they wanted to punish sites that used code to change markup at render time. At least, when I used to have to do onsite SEO work, it was always about simple tag hierarchies. I wonder whether there isn't some better metric of validity and information quality than what markup is used. Some of the sites that surfaced further down could be considered interesting and valuable resources. I think not punishing simple wall-of-text content is a good thing. But to punish more complicated layouts may have the perverse effect of downranking higher-quality sources of information - i.e. people and organizations who can afford to build a decent website, or who care to migrate to a modern blogging platform.
    
    
    
        
      
 I don’t want my search engine to somehow try to judge the believability of the results. I’d like to be the judge of that myself.
A search engine that favors text-heavy sites and punishes modern web design
https://news.ycombinator.com/item?id=28550764
https://www.runnaroo.com/blog/the-search-engine-hacker-news-has-been-asking-for
The Search Engine Hacker News Has Been Asking For
None
None
<div><div>
 <br />
 
 <hr />
 <p>If you read this title and think “I’m on Hacker News and I have never once complained about a search engine” or you feel that {insert your favorite search engine} has never let you down, then this article is probably not relevant and you can save some time and stop reading here.</p>
 <p>For those of you who have voiced concerns on Hacker News or have been quietly disappointed with the current search engines available then there is another option (tl;dr try <a href="https://www.runnaroo.com/">Runnaroo</a>).</p>
 <p>As the most dominant player in the search space, Google receives the brunt of user criticisms, but it is still considered by most to be the best search engine currently available.</p>
 <p>That doesn't mean it is perfect and can't be improved upon though. From an unscientific study done through Hacker News lurking, the most common criticisms of Google fall within the three below groups.</p>
 <ol>
 <li>Privacy and tracking concerns</li>
 <li>Degradation of search result relevance, specifically around Google not respecting exact search queries</li>
 <li>Cluttered search result pages (SERP) that contain difficult to distinguish ads and an increased focus on shopping related results</li>
 </ol>
 <p>If you have ever felt any of the above then I would like to introduce you to Runnaroo.</p>
 <p>Like <a href="https://duckduckgo.com/">DuckDuckGo</a>, which was an early and continual champion on the benefits of privacy for search engines, Runnaroo does not track user behavior to serve targeted search results or ads. </p>
 <p>For users looking for exact term searches, quotation marks are respected for organic web results. There is even a separate “Strict Search” option to ensure that every word in the search query is returned in every result.</p>
 <p>The Runnaroo SERP can contain a number of deep information sources, but always provides a clean and uncluttered layout with any ads begin clearly highlighted and labeled.</p>
 <p>In the immortal words of <a href="https://www.runnaroo.com/search?term=LeVar+Burton">LeVar Burton</a>, "but you don't have to take my word for it."</p>
 <p>You can try it <a href="https://www.runnaroo.com/">here</a>.</p>
 </div>||||I|||| * Start Searching!


 The Search Engine Hacker News Has Been Asking For

 If you read this title and think “I’m on Hacker News and I have never once complained about a search engine” or you feel that {insert your favorite search engine} has never let you down, then this article is probably not relevant and you can save some time and stop reading here.

 For those of you who have voiced concerns on Hacker News or have been quietly disappointed with the current search engines available then there is another option (tl;dr try Runnaroo).

 As the most dominant player in the search space, Google receives the brunt of user criticisms, but it is still considered by most to be the best search engine currently available.

 That doesn't mean it is perfect and can't be improved upon though. From an unscientific study done through Hacker News lurking, the most common criticisms of Google fall within the three below groups.

 1. Privacy and tracking concerns
 2. Degradation of search result relevance, specifically around Google not respecting exact search queries
 3. Cluttered search result pages (SERP) that contain difficult to distinguish ads and an increased focus on shopping related results

 If you have ever felt any of the above then I would like to introduce you to Runnaroo.

 Like DuckDuckGo, which was an early and continual champion on the benefits of privacy for search engines, Runnaroo does not track user behavior to serve targeted search results or ads.

 For users looking for exact term searches, quotation marks are respected for organic web results. There is even a separate “Strict Search” option to ensure that every word in the search query is returned in every result.

 The Runnaroo SERP can contain a number of deep information sources, but always provides a clean and uncluttered layout with any ads begin clearly highlighted and labeled.

 In the immortal words of LeVar Burton, "but you don't have to take my word for it."

 You can try it here.

 About | Privacy | Questions or feedback? Reach us at contact@runnaroo.com
If you read this title and think “I’m on Hacker News and I have never once complained about a search engine” or you feel that {insert your favorite search engine} has never let you down, then this article is probably not relevant and you can save some time and stop reading here.
 For those of you who have voiced concerns on Hacker News or have been quietly disappointed with the current search engines available then there is another option (tl;dr try Runnaroo).
 As the most dominant player in the search space, Google receives the brunt of user criticisms, but it is still considered by most to be the best search engine currently available.
 That doesn't mean it is perfect and can't be improved upon though. From an unscientific study done through Hacker News lurking, the most common criticisms of Google fall within the three below groups.
 
 Privacy and tracking concerns
 Degradation of search result relevance, specifically around Google not respecting exact search queries
 Cluttered search result pages (SERP) that contain difficult to distinguish ads and an increased focus on shopping related results
 
 If you have ever felt any of the above then I would like to introduce you to Runnaroo.
 Like DuckDuckGo, which was an early and continual champion on the benefits of privacy for search engines, Runnaroo does not track user behavior to serve targeted search results or ads. 
 For users looking for exact term searches, quotation marks are respected for organic web results. There is even a separate “Strict Search” option to ensure that every word in the search query is returned in every result.
 The Runnaroo SERP can contain a number of deep information sources, but always provides a clean and uncluttered layout with any ads begin clearly highlighted and labeled.
 In the immortal words of LeVar Burton, "but you don't have to take my word for it."
 You can try it here.
The Search Engine Hacker News Has Been Asking For
https://www.runnaroo.com/blog/the-search-engine-hacker-news-has-been-asking-for
https://thenextweb.com/news/hacker-news-banned-google-search-engine
Hacker News Just Banned Google And Every Other Search Engine [Updated]
2010-03-16
Alex Wilhelm
<div><div>
 <article>
 <div>
 <figure>
 
 
 </figure>
 </div>
 
 
 
 
 
 
 
 
 <main>
 <header>
 <a href="/author/alexwillhelm">
 
 
 </a>
 <div>
 <h6>
 Story by
 </h6>
 <h4>
 <a href="/author/alexwillhelm">
 Alex Wilhelm
 </a>
 </h4>
 
 <p>
 
 
 Alex Wilhelm is a San Francisco-based writer. You can find Alex on Twitter, and on Facebook. You can reach Alex via email at <a href="/cdn-cgi/l/email-protection">[email protected]</a>
 
 (show all)
 
 
 
 Alex Wilhelm is a San Francisco-based writer. You can find Alex on <a href="https://www.twitter.com/alex">Twitter,</a> and on <a href="https://www.facebook.com/alex.wilhelm">Facebook.</a> You can reach Alex via email at <a href="/cdn-cgi/l/email-protection#20414c4558605448454e4558545745420e434f4d">[email protected]</a>
 
 </p>
 </div>
 </header>
 <div>
 <p>Talk about making a point, <a href="https://news.ycombinator.com/">Hacker News</a> (a service of <a href="https://ycombinator.com/">YCombinator</a>) has just blocked every search engine from indexing their website. Want proof? Take a look <a href="https://twitter.com/newsycombinator/status/10541958397">at this</a>:</p>
<figure></figure>
<p>By blocking robots spidering their website, Google (et al) cannot index their content, meaning that if you search for it, you will not find it. Bad news? Not to Hacker News.</p>
<p>Hacker News has long been a tight knit community, and not one driven by page views. They care little if they grow to be as large as Digg, and this move is a strong one to keep their community unique, and cohesive. However, some HN users are <a href="https://news.ycombinator.com/item?id=1194502">not happy</a>:</p>
<figure></figure>
<p>Proof from Hacker News itself:</p>
<figure></figure>
<p>What do you think?</p>
<p><strong>Updated</strong></p>
<p>Paul Graham has <a href="https://news.ycombinator.com/item?id=1194797">spoken</a> and it’s apparently all one misunderstanding:</p>
<p><em>“Don’t worry, it doesn’t mean anything. The software for ranking applications runs on the same server, and it is horribly inefficient (something 4 people use every 6 months doesn’t tend to get optimized much). This weekend all of us were reading applications at the same time, and the system was getting so slow that I banned crawlers for a bit to buy us some margin. (Traffic from crawlers is much more expensive for us than traffic from human users, because it interacts badly with lazy item loading.) We only finished reading applications an hour before I had to leave for SXSW, so I forgot to set robots.txt back to the normal one, but I just did now”</em></p>
<p></p>
 </div>
 
 
 
 
 
 </main>
 
 </article>
 </div>||||I|||| News

 * News
 * Events
 + TNW Conference 2023 June 16 & 17, 2022
 + TNW València 2023
 + All events
 * Spaces
 * Programs
 * Newsletters
 * Partner with us
 * 
 * 
 * 
 * 
 * 
 * 
 * Jobs
 * Contact
 News news news news
 * Latest
 * Data and security
 * Deeptech
 * Ecosystems
 * Fintech and ecommerce
 * Future of work
 * Sustainability
 * More
 + Startups and technology

 The heart of tech is coming to the heart of the Mediterranean. Join TNW in València this March 🇪🇸 →

 This article was published on March 16, 2010

 * Google

 Hacker News Just Banned Google And Every Other Search Engine [Updated]


 March 16, 2010 - 2:24 am

 Story by

 Alex Wilhelm

 Story by

 Alex Wilhelm

 Alex Wilhelm is a San Francisco-based writer. You can find Alex on Twitter, and on Facebook. You can reach Alex via email at [email protected] (show all) Alex Wilhelm is a San Francisco-based writer. You can find Alex on Twitter, and on Facebook. You can reach Alex via email at [email protected]

 Talk about making a point, Hacker News (a service of YCombinator) has just blocked every search engine from indexing their website. Want proof? Take a look at this:

 By blocking robots spidering their website, Google (et al) cannot index their content, meaning that if you search for it, you will not find it. Bad news? Not to Hacker News.

 Discover the future of tech!

 Join us at TNW Conference June 15 & 16 in Amsterdam

 Join now

 Hacker News has long been a tight knit community, and not one driven by page views. They care little if they grow to be as large as Digg, and this move is a strong one to keep their community unique, and cohesive. However, some HN users are not happy:

 Proof from Hacker News itself:

 What do you think?

 Updated

 Paul Graham has spoken and it’s apparently all one misunderstanding:

 “Don’t worry, it doesn’t mean anything. The software for ranking applications runs on the same server, and it is horribly inefficient (something 4 people use every 6 months doesn’t tend to get optimized much). This weekend all of us were reading applications at the same time, and the system was getting so slow that I banned crawlers for a bit to buy us some margin. (Traffic from crawlers is much more expensive for us than traffic from human users, because it interacts badly with lazy item loading.) We only finished reading applications an hour before I had to leave for SXSW, so I forgot to set robots.txt back to the normal one, but I just did now”

 Published March 16, 2010 - 2:24 am UTC
 Back to top


 Popular on TNW

 1. 1

 Europe plots to replace natural gas with geothermal energy

 2. 2

 What Greek myths can teach us about the dangers of AI

 3. 3

 This is the ‘world’s first’ 100% cultivated steak fillet. Fancy a bite?

 4. 4

 6 ways EU startups can cut spending during the recession

 5. 5

 Why TensorFlow for Python is dying a slow death

 Related Articles

 neural

 Can AI design better streets for pedestrians? You be the judge

 house of talent

 7 successful product development strategies team leads can steal from big tech companies

 Join the fiesta in València!

 TNW is coming to València in March, join us now →

 Discover more

 tech

 Google will delete abortion clinic location history — but don’t relax just yet

 plugged

 Google kills Hangouts — but its dreams of Gmail dethroning Slack feel far away

 The heart of tech

 * 
 * 
 * 
 * 
 * 
 * 
 * 


 More TNW

 * Media
 * Events
 * Programs
 * Spaces
 * Newsletters
 * Deals

 About TNW

 * Partner with us
 * Jobs
 * Terms & Conditions
 * Cookie Statement
 * Privacy Statement
 * Editorial Policy
 * Masthead

 Copyright © 2006—2023, The Next Web B.V. Made with <3 in Amsterdam.

Story by
 
 
 
 Alex Wilhelm
 
 
 
 
 
 
 Alex Wilhelm is a San Francisco-based writer. You can find Alex on Twitter, and on Facebook. You can reach Alex via email at [email protected]
 
 (show all)
 
 
 
 Alex Wilhelm is a San Francisco-based writer. You can find Alex on Twitter, and on Facebook. You can reach Alex via email at [email protected]
 
 
 
 
 
 Talk about making a point, Hacker News (a service of YCombinator) has just blocked every search engine from indexing their website. Want proof? Take a look at this:
 
By blocking robots spidering their website, Google (et al) cannot index their content, meaning that if you search for it, you will not find it. Bad news? Not to Hacker News.
Hacker News has long been a tight knit community, and not one driven by page views. They care little if they grow to be as large as Digg, and this move is a strong one to keep their community unique, and cohesive. However, some HN users are not happy:
 
Proof from Hacker News itself:
 
What do you think?
 Updated 
Paul Graham has spoken and it’s apparently all one misunderstanding:
 “Don’t worry, it doesn’t mean anything. The software for ranking applications runs on the same server, and it is horribly inefficient (something 4 people use every 6 months doesn’t tend to get optimized much). This weekend all of us were reading applications at the same time, and the system was getting so slow that I banned crawlers for a bit to buy us some margin. (Traffic from crawlers is much more expensive for us than traffic from human users, because it interacts badly with lazy item loading.) We only finished reading applications an hour before I had to leave for SXSW, so I forgot to set robots.txt back to the normal one, but I just did now”
Hacker News Just Banned Google And Every Other Search Engine [Updated]
https://thenextweb.com/news/hacker-news-banned-google-search-engine
https://stackoverflow.com/questions/8660344/how-does-google-chrome-know-to-use-hnsearch-to-search-hacker-news
How Does Google Chrome know to use HNSearch to search Hacker News?
2011-12-28
Zameer Manji Zameer Manji 2
<div><div>
 
<div>
 
 
<div>
 
 <div>
 
<p>Hacker News' URL is news.ycombinator.com. When I input the full URL into Chrome, the right most part of the URL bar has the text "Press to search HNSearch". HNSearch is a separate site which indexes and searches Hacker News. It is located at hnsearch.com. There is nothing in the metadata of Hacker News to indicate that HNSearch is the search engine of Hacker News.</p>
<p>So my question is, How does Google Chrome know to use HNSearch to search Hacker News?</p>
 </div>
 
 <div>
 <p>
 asked Dec 28, 2011 at 19:29
 </p>
 <div>
 <a href="/users/2874/zameer-manji"><p></p></a>
 </div>
 <div>
 <p><a href="/users/2874/zameer-manji">Zameer Manji</a>Zameer Manji</p><p>2,9875 gold badges30 silver badges42 bronze badges
 </p>
 </div>
</div>
 
</div>
 <p>1</p>
 </div>
 <div>
 
 
<div>
 
 
<div>
 
 <p>Likely Chrome is asking Google for what search forms are available. The search form is using hnsearch.com.</p>
 <div>
 <p>
 answered Dec 28, 2011 at 19:37
 </p>
 <div>
 <a href="/users/23528/daniel-a-white"><p></p></a>
 </div>
 <div>
 <p><a href="/users/23528/daniel-a-white">Daniel A. White</a>Daniel A. White</p><p>185k46 gold badges364 silver badges438 bronze badges
 </p>
 </div>
</div>
 
</div>
 <p>1</p>
 </div>
 
 <h2>
 
 </h2>
 </div>
 </div>||||I|||| Stack Overflow
 1. About
 2. Products
 3. For Teams
 1. Stack Overflow Public questions & answers
 2. Stack Overflow for Teams Where developers & technologists share private knowledge with coworkers
 3. Talent Build your employer brand
 4. Advertising Reach developers & technologists worldwide
 5. About the company
 Loading…

 1. current community

 + Stack Overflow
 help chat
 + Meta Stack Overflow

 your communities

 Sign up or log in to customize your list.

 more stack exchange communities

 company blog
 2. 
 3. Log in
 4. Sign up
 1. Home
 2. 
 1. Public
 2. Questions
 3. Tags
 4. Users
 5. Companies
 6. Collectives
 7. Explore Collectives
 3. 
 1. Teams
 Stack Overflow for Teams – Start collaborating and sharing organizational knowledge. Create a free Team Why Teams?
 2. Teams
 3. Create free Team

 Collectives™ on Stack Overflow

 Find centralized, trusted content and collaborate around the technologies you use most.

 Learn more about Collectives

 Teams

 Q&A for work

 Connect and share knowledge within a single location that is structured and easy to search.

 Learn more about Teams

 How Does Google Chrome know to use HNSearch to search Hacker News?

 Ask Question
 Asked 11 years, 1 month ago
 Modified 11 years, 1 month ago
 Viewed 195 times
 1

 Hacker News' URL is news.ycombinator.com. When I input the full URL into Chrome, the right most part of the URL bar has the text "Press to search HNSearch". HNSearch is a separate site which indexes and searches Hacker News. It is located at hnsearch.com. There is nothing in the metadata of Hacker News to indicate that HNSearch is the search engine of Hacker News.

 So my question is, How does Google Chrome know to use HNSearch to search Hacker News?

 * google-chrome
 * metadata
 Share
 Improve this question
 Follow
 asked Dec 28, 2011 at 19:29
 Zameer Manji Zameer Manji
 2,987 5 5 gold badges 30 30 silver badges 42 42 bronze badges
 1
 * Here you go: possible duplicate of How do search engines recognize search boxes on websites? — no metadata, Chrome just "guesses" when you use a form.
 – Matt Ball
 Dec 28, 2011 at 19:36
 Add a comment |

 1 Answer

 Sorted by: Reset to default
 Highest score (default) Trending (recent votes count more) Date modified (newest first) Date created (oldest first)
 0

 Likely Chrome is asking Google for what search forms are available. The search form is using hnsearch.com.

 Share
 Improve this answer
 Follow
 answered Dec 28, 2011 at 19:37
 Daniel A. White Daniel A. White
 185k 46 46 gold badges 364 364 silver badges 438 438 bronze badges
 1
 * How does Google know what search forms are available? Is there no metadata involved? How can it guess the search form for Hacker News but not for Stack Overflow?
 – Zameer Manji
 Dec 28, 2011 at 20:39
 Add a comment |

 Your Answer

 Thanks for contributing an answer to Stack Overflow!

 * Please be sure to answer the question. Provide details and share your research!

 But avoid …

 * Asking for help, clarification, or responding to other answers.
 * Making statements based on opinion; back them up with references or personal experience.

 To learn more, see our tips on writing great answers.

 Sign up or log in

 Sign up using Google
 Sign up using Facebook
 Sign up using Email and Password
 Submit

 Post as a guest

 Name
 Email

 Required, but never shown

 Post as a guest

 Name
 Email

 Required, but never shown

 Post Your Answer Discard

 By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

 Not the answer you're looking for? Browse other questions tagged
 * google-chrome
 * metadata
 or ask your own question.

 * The Overflow Blog
 * Why developer experience is the key to better software, straight from the...
 * The frontier of computing: Q&A with IBM quantum computing research
 * Featured on Meta
 * Accessibility Update: Colors
 * 2022: a year in moderation
 * Should we burninate the [balance] tag?
 * Collectives: The next iteration
 * Temporary policy: ChatGPT is banned

 Linked

 1
 How do search engines recognize search boxes on websites?

 Related

 459
 How to search all loaded scripts in Chrome Developer Tools?
 1412
 How to manually send HTTP POST requests from Firefox or Chrome browser
 172
 How to add google chrome omnibox-search support for your site?
 1220
 How do I get ASP.NET Web API to return JSON instead of XML using Chrome?
 751
 How to set a JavaScript breakpoint from code in Chrome?
 25
 How to let Google detect a site search engine?
 526
 Google Chrome redirecting localhost to https

 Hot Network Questions

 * Did JK Rowling ever describe Harry Potter goblins as "hook-nosed"?
 * I forgot to include $10 in interest in my tax return. My tax liability did not change. Should I file an amended return?
 * Addition and multiplication are commutative but exponentiation and tetration are not. Do we know why?
 * GFCI Light Is On, Test Button Doesn't Work
 * Why is b.pop(0) over 200 times slower than del b[0] for bytearray?
 * How to adapt a spoke salvaged from a 26" wheel to a 24" one?
 * If a balloon lies within gun range, why would a 21st century warplane shoot it with a missile, instead of rotary guns?
 * Why are there 23 members of the grand jury?
 * Help with Graph Consisting of Lines and Nodes
 * Can a creature be targeted by future uses of eyebite if they are immune to the initial casting?
 * Why does Ontario have a quota on milk production?
 * Better midi playback on mac
 * How to deal with a professor in the department undermining my field of research to colleagues and other professors?
 * How has the resolution of astronomical spectrographs improved over time?
 * Table of unequal sequence
 * Stable words in the Proto-Indo-European language family
 * Determining the optimize lambda in Multi-Objective Optimization
 * What changes to Voyager could have been made?
 * Understanding how to terminate recurrence relations in quantum SHO
 * Crashing the sha1() function in PHP?
 * At what point does combining integrated circuits yield something patentable?
 * First Age "of the Sun" terminology
 * Find the perimeter (seemingly unsolvable problem)
 * What is the use of alumium profile with LED strip lighting?
 more hot questions
 Question feed

 Subscribe to RSS

 Question feed

 To subscribe to this RSS feed, copy and paste this URL into your RSS reader.

 Stack Overflow

 * Questions
 * Help

 Products

 * Teams
 * Advertising
 * Collectives
 * Talent

 Company

 * About
 * Press
 * Work Here
 * Legal
 * Privacy Policy
 * Terms of Service
 * Contact Us
 * Cookie Settings
 * Cookie Policy

 Stack Exchange Network

 * Technology
 * Culture & recreation
 * Life & arts
 * Science
 * Professional
 * Business
 * API
 * Data
 * Blog
 * Facebook
 * Twitter
 * LinkedIn
 * Instagram

 Site design / logo © 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA. rev 2023.2.6.43221

 Your privacy

 By clicking “Accept all cookies”, you agree Stack Exchange can store cookies on your device and disclose information in accordance with our Cookie Policy.

 Accept all cookies Customize settings
Hacker News' URL is news.ycombinator.com. When I input the full URL into Chrome, the right most part of the URL bar has the text "Press to search HNSearch". HNSearch is a separate site which indexes and searches Hacker News. It is located at hnsearch.com. There is nothing in the metadata of Hacker News to indicate that HNSearch is the search engine of Hacker News.
So my question is, How does Google Chrome know to use HNSearch to search Hacker News?
 
 
 
 
 asked Dec 28, 2011 at 19:29
 
 
    
 
 
  Zameer ManjiZameer Manji 2,9875 gold badges30 silver badges42 bronze badges
 
 
 
 1
 
 
 
 
 
 
 
 Likely Chrome is asking Google for what search forms are available. The search form is using hnsearch.com.
 
 
 answered Dec 28, 2011 at 19:37
 
 
    
 
 
  Daniel A. WhiteDaniel A. White 185k46 gold badges364 silver badges438 bronze badges
 
 
 
 1
How Does Google Chrome know to use HNSearch to search Hacker News?
https://stackoverflow.com/questions/8660344/how-does-google-chrome-know-to-use-hnsearch-to-search-hacker-news